NimbleOS 5.1.1.0 Release Notes - Hewlett Packard Enterprise...NimbleOS 5.1.1.0 Version: 5.1.1.0...

NimbleOS 5.1.1.0 ReleaseNotes

Version 5.1.1.0

Published November, 2019

© Copyright 2019 Hewlett Packard Enterprise Development LP. All rights reserved worldwide.

Notices

The information contained herein is subject to change without notice. The only warranties for Hewlett PackardEnterprise products and services are set forth in the express warranty statements accompanying such productsand services. Nothing herein should be construed as constituting an additional warranty. Hewlett Packard Enterpriseshall not be liable for technical or editorial errors or omissions contained herein.

Confidential computer software. Valid license from Hewlett Packard Enterprise required for possession, use, orcopying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documen-tation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor's standardcommercial license.

Links to third-party websites take you outside the Hewlett Packard Enterprise website. Hewlett Packard Enterprisehas no control over and is not responsible for information outside the Hewlett Packard Enterprise website.

Acknowledgments

Intel®, Itanium®, Pentium®, Intel Inside®, and the Intel Inside logo are trademarks of Intel Corporation in theUnited States and other countries.

Microsoft® and Windows® are either registered trademarks or trademarks of Microsoft Corporation in the UnitedStates and/or other countries.

Adobe® and Acrobat® are trademarks of Adobe Systems Incorporated. Java® and Oracle® are registered trade-marks of Oracle and/or its affiliates.

UNIX® is a registered trademark of The Open Group.

Publication Date

Friday November 22, 2019 04:20:32

Document ID

oba1551820136514

Support

All documentation and knowledge base articles are available on HPE InfoSight at https://infosight.hpe.com.To register for HPE InfoSight, click the Create Account link on the main page.

Email: [email protected]

For all other general support contact information, go to https://www.nimblestorage.com/customer-support/.

2Copyright © 2019 by Hewlett Packard Enterprise Development LP. All rights reserved.

https://infosight.hpe.com

mailto:[email protected]

https://www.nimblestorage.com/customer-support/

Contents

NimbleOS 5.1.1.0..........................................................................................4Important Update Note.........................................................................................................................4

Special Notes.......................................................................................................................................4

New Features in 5.1.1.0.......................................................................................................................9

Verified Update Paths........................................................................................................................10

Known Critical Issues.........................................................................................................................13Resolved Critical Issues.....................................................................................................................17Resolved Issues.................................................................................................................................22Known Issues.....................................................................................................................................35

Copyright © 2019 by Hewlett Packard Enterprise Development LP. All rights reserved.

Nimble

NimbleOS 5.1.1.0

5.1.1.0Version:

Wednesday July 17, 2019 14:37:32Revision:

The release notes describe the major changes, fixes, and known issues for this release of the NimbleOS.They do not include all individual fixes and internal changes.

For technical support, contact HPE Nimble Storage Support at:

mailto:[email protected] (877-364-6253), option 2.

Important Update NoteUpdating NimbleOS can involve an update to component firmware on the standby controller. This can causean email alert and automated case indicating "Standby Controller Not Available" when the firmware updateprocess takes longer than five minutes. This is expected behavior and does not affect data services. At theend of the software update, you can check status of both controllers in the Web UI under Manage > Hardware.One controller will be ACTIVE and the other STANDBY under normal operating conditions following asuccessful software update.

All third-party software notices can be found on HPE InfoSight (https://infosight.hpe.com) on the Resources >Documentation page:

https://infosight.hpe.com/resources/nimble/docs

The Documentation page also includes the General Terms and Conditions document. You can display thisdocument by performing the following steps:

1 In the navigation pane on the HPE InfoSight Documentation page, scroll through the Document Typelist and select Support Policy.

2 In the page that appears, select General Terms and Conditions. This document opens in a browsertab.

Special Notes

DescriptionNote

HPE Nimble Storage continues to qualify configurations between releas-es. The Validated Configuration Matrix provides information about vali-dated configurations and is updated frequently. It is a good practice tocheck your system configuration against this online tool. The ValidatedConfiguration Matrix tool is available on HPE InfoSight:

https://infosight.hpe.com/resources/nimble/validated-configuration-matrix

CRITICAL

Arrays must be running NimbleOS 5.0.4.0 or later to upgrade to Nim-bleOS 5.1.1.0.

CRITICAL

Internet Explorer 10 and earlier versions are not supported in NimbleOS4.x and later.

CRITICAL


NimbleOS 5.1.1.0 Important Update Note

mailto:[email protected]

https://infosight.hpe.com



DescriptionNote

An extended data services outage may occur with MS iSCSI initiatorand Intel NICs using the built-in Windows driver e1q60x64.sys (version11.0.5.21/11.0.5.22).

If you encounter this problem, please update your system to use thelatest Windows driver.

CRITICAL

A service outage may occur on Windows 2012 R2 hosts using Emulexor Broadcom Fibre Channel HBAs with firmware/driver prior to 11.2.Update the Emulex or Broadcom firmware/driver to 11.2 or later

CRITICAL

Due to a known Red Hat Enterprise Linux bug 1002727, while runningvirtualized in VMware ESX, manually rebooting the active controller inpresence of heavy IOs using the reboot --controller command on aFibre Channel array may trigger an incorrect retry initiated by RHELguests running the following kernel versions:

• 6.4 and earlier• 6.5 without the patch• 7.0 without the patch

This incorrect retry logic may lead to unexpected application behavior.In these environments, we recommend the failover command instead.

CRITICAL

Due to a known Red Hat Enterprise Linux bug 3550561, unexpectedapplication behavior may occur on RHEL 7.5 hosts with kernel-3.10.0-862.3.2.el7 or derivatives using Emulex FC FCoE HBAs (lpfc driver)and raw devices. To avoid this issue:

• If running RHEL 7.6, update to kernel-3.10.0-957.el7 or later.• If running RHEL 7.5z, update to kernel-3.10.0-862.25.3.el7 or later.

CRITICAL

As outlined in the current Validated Configuration Matrix, HPE NimbleStorage fully supports Windows guest operating systems on MicrosoftHyper-V, including Virtual Fibre Channel (VFC) connectivity and multi-pathing with HPE Nimble Storage DSM and VSS support. However,Linux guest operating systems running in Hyper-V VFC configurationsare not qualified. Running Red Hat Linux guest operating systems withthe “Linux Integration Services” kit installed, or with hv_storvsc driversin such configurations can lead to Red Hat bug 1364282, which cancause an unexpected service outage.

CRITICAL

Starting with NimbleOS 5.1.1.0, the size of the software package nowexceeds 2 GB, which may lead to lengthier software download times.Previously, the sizes of the NimbleOS 5.0.x download packages wereapproximately 1.6 GB, and NimbleOS 4.x packages were approximately900 MB.

Important

HPE Nimble Storage CSxx (except CS235) and CS4xx arrays areblocked from updating to NimbleOS 5.1.1.0 and later releases by default.These arrays will be allowed to update in special circumstances, suchas for group merge and evacuation purposes, and temporary data mi-gration workloads. Contact HPE Nimble Storage Support if you wish toupdate your CS2xx or CS4xx array to NimbleOS 5.1.x.x.

Important

Microsoft Offload Data Transfer (ODX) is not supported if the destinationvolume has synchronous replication enabled.

Important


NimbleOS 5.1.1.0 Special Notes

DescriptionNote

Starting with NimbleOS version 5.0.7.0, the Fibre Channel HBAs willuse an updated firmware (11.4.204). The new firmware addresses anissue in which some 16Gb Fibre Channel HBAs might not auto-negotiateto 16Gb on all ports due to a timing issue within the code of the previousversion of the firmware.

Important

As of vSphere 6.5, VMware is discontinuing the Thick Client (also knownas the desktop or C# Client). As a result, the HPE Nimble StoragevCenter Plugin is deprecating the Thick Client and future releases ofNimbleOS will not support it.

Important

Starting with version 5.0.3.0, NimbleOS includes a restriction that pre-vents you from enabling deduplication when you are using a CS3000,CS5000, CS7000, HF20, HF40, or HF60 array and have fewer than sixSSDs. This restriction is necessary to prevent the possibility of significantperformance issues.

Because NimbleOS 5.0.2.0 and 5.0.1.0 did not enforce this restriction,arrays upgrading from those releases may already have volumes withdeduplication enabled. Any array upgrading to 5.0.3.0 or later withdeduplicated volumes will continue to operate as a dedupe capablearray, regardless of the number of installed SSDs. Such configurationsare not recommended by HPE Nimble Storage.

The following table lists the number of SSDs required for the differentarrays:

Required Number of SSDsArray Model

2 SSDsHF20H

4 SSDsHF20H upgraded to full population

4 SSDsHF20H fully populated and upgrad-ed to HF40H

6 SSDsHF20, HF40, HF60

4 SSDsCS500

4 SSDsCS700

6 SSDsCS3000

6 SSDsCS5000

6 SSDsCS7000

Important



DescriptionNote

You can enable deduplication for CS3000, CS5000, CS7000, CS700,and CS500 arrays on a volume only if the corresponding storage poolhas a Flash to Disk Ratio (FDR) greater than 4%. To calculate the FDR,obtain the "Total array capacity (MiB)" and "Total array cache capacity(MiB)" values by using the HPE Nimble Storage CLI command pool --info pool_name. This command returns the Pool capacity (MiB), whichis the "Total array capacity (MiB)", and the Pool cache capacity (MiB),which is the "Total array cache capacity (MIB)".

Then perform the following calculation:

FDR = "Total array cache capacity (MiB)"/"Total array capacity(MiB)" * 100

If the array has sufficient capability for deduplication, the pool --infocommand will also show a value for dedupe capacity (MiB).

Note On the HF20H, HF20, HF40, and HF60 platforms, pool --infodisplays "N/A" as the value for dedupe capacity (MiB). This becauseyou can enable deduplication for the entire array.

Important

For connections to the NimbleOS GUI, you must have port 5392 openfor the Group Management IP address and both diagnostic IP addresses.

Important

VMware has announced End of General Support for vSphere 5.0 andvSphere 5.1, which includes vSphere Hypervisor ESXi 5.0 and 5.1 andvCenter Server 5.0 and 5.1. To maintain your full level of support andsubscription from VMware, you should upgrade to a supported versionof vSphere. Refer to the VMware Knowledge Base article KB2145103.

Important

During deployment of a desktop using VMware Horizon View, a VVol(mapping to a disposable disk) is created. A clone of this VVol is alsocreated and placed within a directory under the virtual machine directorynamed sdd.

When this desktop is deleted from Horizon View, VMware fails to deletethe VVol clone of the disposable disk; only the disposable disk itself isdeleted. This will be fixed in the next vSphere release. VMware bugnumber 1807857 should be used to track this fix.

This issue occurs on all versions of VMware Horizon and vSphere thatsupport VVols:

• Horizon 6 version 6.1 and later• vSphere 6.0 and later

Important

On Windows Server 2012, 2012 R2, and 2016, the disk optimizationprocess may record the following error in the Application event log: "Thevolume was not optimized because an error was encountered: NeitherSlab Consolidation nor Slab Analysis will run if slabs are less than 8MB. (0x8900002D)". Although Windows records this as an Error in theevent log, the event can be safely ignored for HPE Nimble Storagevolumes. HPE Nimble Storage volumes do not benefit from or requireslab consolidation.

Important



https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2145103

DescriptionNote

Numerous host integration toolkits are supported in NimbleOS 5.1.1.0.It is strongly recommended that they be installed on all Windows, Linux,and VMware hosts. For more information about supported toolkits, referto the Validated Configuration Matrix, which is available on HPE NimbleStorage InfoSight:


Important

Hosts leveraging VSS integration with HPE Nimble Storage WindowsToolkit (NWT) 2.1 and earlier will not work with NimbleOS 3.x. and later.

Important

When deleting files on a Windows Server 2016 HPE Nimble Storagevolume that was formatted with ReFS, the user may find the space isnot freed from the array. The reason for this is that Microsoft has chosento leave trim disabled by default on ReFS v2 volumes. This is detailedin the following Microsoft technet article:

https://technet.microsoft.com/en-us/windows-server-docs/manage-ment/windows-commands/fsutil-behavior

In the "Parameters" list, review the "DisableDeleteNotify" section fordetails. For customers using ReFS v2, on Windows Server 2016,deleted files will not be unmapped by default. You need to enable trimby executing:

fsutil behavior set DisableDeleteNotify ReFS 0

Important

HPE Nimble Storage Connection Manager (NCM) for VMware 5.1.0 issigned by VMware for ESXi 5.x and ESXi 6.x. It can be installed throughthe VMware Update Manager or esxcli command without the --no-sig-check flag.

See NCM for VMware Release Notes 5.1.0 or later and the latestVMware Integration Guide for further details.

Important

Performing a group merge from a source group that contains runningHyper-V virtual machines requires additional care. Group merges requirechanges to the discovery IP address that can adversely impact runningsystems. Therefore, if you perform a group merge, you should plan amaintenance outage to gracefully stop all applications and Hyper-Vvirtual machines on the source group to eliminate unexpected downtimecaused by changing IP address during the group merge process. Atypical group merge should take only a few minutes to complete andthen virtual machines and applications can be restarted.

The group merge and pool merge operations will also have impact onSCVMM. The impact will depend on whether the source and destinationgroups or pools are under SCVMM's management.

Please refer to the SMI-S Integration Guide which includes details aboutSCVMM and the impacts in these situations before performing mergeoperations.

Important




https://technet.microsoft.com/en-us/windows-server-docs/management/windows-commands/fsutil-behavior

https://technet.microsoft.com/en-us/windows-server-docs/management/windows-commands/fsutil-behavior

https://infosight.hpe.com/InfoSight/media/cms/active/pubs_VMware_Integration_Guide.pdf

DescriptionNote

Various timeout values affect HPE Nimble Storage targets from Win-dows/Linux hosts. Before you update the NimbleOS, install the HPE Nimble Storage Windows Toolkit (NWT) or HPE Nimble Storage Linux Toolkit (NLT) on the host or tune the timeout values. Timeout details for various operating systems can be found on HPE InfoSight under Resources > Documentation. From the HPE Nimble Storage Docu-mentation page, select Knowledge Base Article and choose the KB you want:

• KB-000052: Windows Host Disk Timeout Values• KB-000087: VMware Host Disk Timeout Values• KB-000304: Linux Host Disk Timeout Values

Important

The Backup Repository performance policy introduced in NimbleOS 4.2 cannot be used when replicating against a downstream array running an older release. Replicated volumes need to be associated at the time of creation with a performance policy that either exists downstream or that can be manually created on the downstream array.

Important

New Features in 5.1.1.0Synchronous Replication

This feature provides the ability to synchronously replicate data between two arrays located in separate datacenters. Synchronous replication provides automatic protection against array or site failure. Only supported on AFxx, HFxx, AFxxxx and CSxxxx and newer arrays.

Manual Group Leader Failover

This feature supports the ability to move the group leader functionality from one array to another when arraysare grouped. This allows the retirement of the array supporting the group leader functionality.

Group Scoped iSCSI Target

This feature allows multiple LUNs to be accessed through a single iSCSI target, which reduces the numberof connections required when configuring a large number of LUNs.

Space Reporting Changes

This feature provides simplified space reporting when using data reduction tools, such as compression anddeduplication. Arrays will now show logical mapped usage, which more closely aligns with HPE InfoSightand hosts in terms of per-reduction space usage reported at the volume level. In addition, NimbleOS willactually enforce limits, and introduces the concept of overdraft, which allows a folder to exceed the provisionedlimit before the limit is enforced. Going forward, reserves will only be thin (0%) or thick (100%). Volumequotas become volume limits. Snapshot quotas no longer exist.

Folder Level Space Enforcement

Expanded the functionality for enforcing space usage within individual folders.

Multi-protocol (iSCSI/FC) Access to Same Array

This feature allows both the iSCSI and Fibre Channel protocols to be used simultaneously on a single arrayor group to access different LUNs.

Custom Password Management Policies

This feature provides improved password policies to enable greater security for user access to the HPENimble Storage array.

vCenter Plugin HTML5 Client

This feature improves the vCenter plugin by leveraging HTML5 technology to better interact with VMwarevCenter.


NimbleOS 5.1.1.0 New Features in 5.1.1.0

Time-To-Live (TTL)This feature enables the automatic deletion of unmanaged snapshots.



https://infosight.hpe.com/InfoSight/media/kb/active/htr1455131698674.whz/index.html



Verified Update Paths

Table 1: From Versions 5.x

From Versions 5.x

To VersionFrom Version

5.1.1.05.0.7.200

5.1.1.05.0.7.100

5.1.1.05.0.7.0

5.1.1.05.0.6.0

5.1.1.05.0.5.200

5.1.1.05.0.5.0

5.1.1.05.0.4.0

5.0.7.2005.0.3.100

5.0.7.2005.0.3.0

5.0.7.2005.0.2.0

5.0.7.2005.0.1.100

5.0.7.2005.0.1.0


From Versions 4.x


5.1.1.04.5.4.0

5.0.7.2004.5.3.0

5.0.7.2004.5.2.0

5.0.7.2004.5.1.0

5.0.7.2004.5.0.0

5.0.7.2004.4.1.0

5.0.7.2004.4.0.0

5.0.7.2004.3.1.0

5.0.7.2004.3.0.0

5.0.7.2004.2.1.0

5.0.7.2004.2.0.0

5.0.7.2004.1.0.0


NimbleOS 5.1.1.0 Verified Update Paths


From 3.x Versions


5.0.7.2003.9.0.0

5.0.7.2003.8.1.0

5.0.7.2003.8.0.0

5.0.7.2003.7.0.0

5.0.7.2003.6.2.0

5.0.7.2003.6.1.0

5.0.7.2003.6.0.0

5.0.7.2003.5.4.0

5.0.7.2003.5.3.0

5.0.7.2003.5.2.0

5.0.7.2003.5.0.0

5.0.7.2003.4.1.0

5.0.7.2003.4.0.0

5.0.7.2003.3.0.0

5.0.7.2003.2.1.0

5.0.7.2003.1.0.0


From 2.1.x VersionsFrom 2.1.x VersionsFrom 2.2.x, 2.3.x Versions

To VersionFrom VersionTo VersionFrom VersionTo VersionFrom Version

2.1.9.12.8.0.02.3.18.02.1.9.14.5.4.02.3.18.0

2.1.9.12.0.7.02.3.18.02.1.9.04.5.4.02.3.16.0

2.1.9.12.0.6.*2.3.18.02.1.8.04.5.4.02.3.15.0

2.1.9.12.0.5.02.2.9.02.1.7.04.5.4.02.3.14.0

2.1.9.12.0.4.02.2.9.02.1.6.04.5.4.02.3.12.*

2.2.9.02.1.5.04.5.4.02.3.9.*

2.2.9.02.1.4.04.5.4.02.3.8.0

2.2.9.02.1.3.04.5.4.02.3.7.0

2.2.9.02.1.2.04.5.4.02.3.6.0

2.1.9.12.1.1.04.5.4.02.3.4.0

2.1.9.12.1.0.04.5.4.02.3.3.0

4.5.4.02.3.2.1


NimbleOS 5.1.1.0 Verified Update Paths

From 2.1.x VersionsFrom 2.1.x VersionsFrom 2.2.x, 2.3.x Versions


4.5.4.02.3.2.0

4.5.4.02.3.1.0

3.9.0.02.2.11.0

3.9.0.02.2.10.0

3.9.0.02.2.9.0

3.9.0.02.2.7.*

3.9.0.02.2.6.0

3.9.0.02.2.5.*

2.2.11.02.2.3.*

2.2.11.02.2.2.0

2.2.11.02.2.1.0

2.2.11.02.2.0.0


From 1.0.x VersionsFrom 1.3, 1.2, 1.1 VersionsFrom 1.4.x Versions


Contact Support1.0.7.*1.4.6.01.3.*.*2.1.9.11.4.12.0

Contact Support1.0.6.*1.4.6.01.2.*.*2.1.9.11.4.11.0

1.2.2.01.1.*.*2.1.9.11.4.10.0

2.1.9.11.4.9.0

2.1.9.11.4.8.0

2.1.9.11.4.7.0

1.4.12.01.4.*.*


NimbleOS 5.1.1.0 Known Critical Issues

Known Critical Issues

Known Critical Issues in NimbleOS version 5.1.1.0

WorkaroundDescriptionTitleComponentID

When planning to removea member array from group,schedule a planned mainte-nance window and place allESX hosts into mainte-nance mode to minimizeimpact to availability. ESXtypically resumes connec-tion to VVol datastores, andreconnects to VMs, after aperiod of 15-30 minutes au-tomatically without a manu-al intervention.

Scaled VVol environmentswith 500 VVol VDI VMs ormore than 5000 NimbleVVol volumes may experi-ence IO disruption whenremoving a member arrayfrom group. Symptom ofproblem would appear asVVol datastores being (inac-cessible). Virtual Machinestatus would also appear as(inaccessible).

Removing mem-ber array frommulti-arraygroup maycause IO disrup-tion to scaledVVol environ-ments

Data ServiceAS-77607

Please contact HPE NimbleStorage Support.

Missing IPMI sensors onController B of AFXX andHFXX arrays may lead toincorrect power supply, fanand temperature readings.This may result in falsepower supply, fan, andtemperature alerts followinga controller reboot orfailover.

False powersupply, fan, andtemperaturereadings onController B ofAFXX and HFXXarrays

PlatformAS-95938

None. The controller rebootis enough to restore theHigh Availability state.

The array runs post-mortemanalysis when one of theessential NimbleOS ser-vices terminates abnormal-ly. In rare occasions, suchanalysis may consumeenough memory to trip theout-of-memory threshold, atwhich point, the controlleris rebooted in an attempt torestore normal service. HPENimble Storage is currentlystudying ways to run post-mortem analysis with alower memory footprint im-pact.

Unexpectedcontrollertakeover due toout-of-memorycondition

PlatformAS-95087





Not applicableIn rare occurrences, a cus-tomer-initiated reboot maycause a kernel reboot onthe active controller. Thiswill cause a longer rebootcycle.

Graceful shut-down takeslonger than ex-pected

PlatformAS-95169

Please contact HPE NimbleStorage Support

After updating the array toNimbleOS 5.1.1.0, the ser-vice that sends out eventsand alerts may stop. Thisoccurs if alarms exist on thearray that have been depre-cated in version 5.1.1.0.

Events Servicestops on the ar-ray after updateto 5.1.1.0

System Manage-ment

AS-96050

A Manual Group LeaderFailover will be required torestore Fibre Channel con-nectivity to the hosts.

An Automatic Failover(AFO) of the Group Manage-ment Services will not beinitiated if all Fibre Channel(FC) interfaces on theGroup Leader array fail onboth controllers.

No AutomaticFailover in theevent the hostloses all FC con-nectivity to anarray

System Manage-ment

AS-94737

Reenter passphrase afterAutomatic Failover.

If encrypted volumes areconfigured and AutomaticFailover happens, encrypt-ed volumes wont come on-line automatically after theother array takes overGroup Management ser-vices. The user will need toenter the passphrase onnew group leader array inorder to bring the encryptedvolumes back online.

AutomaticFailover ofGroup Servicesis not supportedfor EncryptedVolumes

System Manage-ment

AS-93553

A failover can be initiated inorder to start the GroupManagement Service. Youmay also contact HPE Nim-ble Storage Support torestart the service manually.

When performing a con-troller upgrade to a high-end model, the object limitswill still show the lower lim-its if the Group Manage-ment Service is not restart-ed.

Need to restartthe Group Man-agement Serviceto unlock addi-tional volumelimits after con-troller upgrade

System Manage-ment

AS-65615





Not applicableWhen attempting to performa pool merge operation, ifthere are a large number ofvolumes that must bestriped across the pool, andone of the arrays has alarge number of pendingdeletes, then it is possiblefor the operation to fail dueto the Data Service beingoverloaded. Symptoms ofthis behavior are if the poolmerge operation hangs forseveral minutes and returnsthe following message: Therequest could not be under-stood by the server.

Pool merge failsdue to too anypending deletes

System Manage-ment

AS-95470

Not applicableIn rare instances, the ArrayManagement service maybecome unavailable formore than 10 minutes on aGroup Leader array after aGroup Leader failover. Afterthe service is unavailablefor ten minutes, active con-troller will perform a failover.The service will recover af-ter controller failover.

Array Manage-ment temporarilyunavailable afterGroup Leaderfailover

System Manage-ment

AS-95642

Not applicableWhile waiting for an underly-ing database migration tocomplete during a Nim-bleOS software update frompre-4.x.x.x to 4.x.x.x+, lowmemory arrays with high IOload may see an unexpect-ed service restart of the Ar-ray Management or GroupManagement service. Ser-vices will automatically re-cover after the restart with-out any impact to the soft-ware update process.

Array servicesmay restartaround comple-tion of softwareupdate to Nim-bleOS 4.x.x.x+

System Manage-ment

AS-80338





There is no workaround. Toavoid, reduce IO load whenperforming software update.

The underlying scale-outdatabase competes withCASL and other systemprocesses for IOPS. Duringsoftware update, a migra-tion script runs against thedatabase. Under heavy filesystem load, the migrationsteps may not completewithin the expected amountof time. As a result, the mi-gration may timeout leadingto a restart of the GroupData Service. After therestart, the migration shouldeventually complete as nor-mal without any user impactor intervention.

Group Data Ser-vice may restartwhen the arrayis under heavyload during soft-ware update

System Manage-ment

AS-81279


In certain situations, informa-tion about the Group Lead-ers SSH key may be mis-configured. This may leadto failures in passwordless-SSH connections betweenthe Group Leader andBackup Group Leader ar-rays. When this happens, itmay take longer to set upthe Backup Group Leader,and in some cases, thegroup setup will go out-of-sync.

Passwordless-SSH connectionfails betweenGroup Leaderand BackupGroup Leader

System Manage-ment

AS-95637



Resolved Critical Issues

Resolved Critical Issues in NimbleOS version 5.1.1.0


Not applicableAs part of the file systemcheckpoint process, internalindexes are required tomerge to disk. Under somecircumstances, the mergesfail to complete within aspecified time when theyare competing with othermerges, for CPU and Diskbandwidth, that are not partof the checkpoint. Whenthat happens, the DataService may restart with ahealth check failure.

Data Servicemay restart unex-pectedly withVM health checkfailure when in-ternal index datastructures taketoo long tomerge


Not applicableIn rare cases duringplanned or unplanned DataService stop/start/restartevent, a deadline check canbe reached before an inter-nal buffer is fully released.As a result, a unexpectedservice restart could occur.The service should automat-ically recover the condition.

Data Servicemay restart unex-pectedly restartwhen the pro-cess is alreadyrestarting


Contact Nimble StorageSupport

The disk Index layer createsnew Index tree for eachcheckpoint for data blockschanged in that checkpoint.Opportunistically thesesmall trees are merged intoa single large tree. Thismerge process might createnew NVRAM records whichneed to be committed as asingle transaction. If thenumber of records is verylarge, time allotted to thetransaction exceeds the in-ternal file system timeoutand Data Service Demoncrashes and restarts itself

ATM Healthcheck failurewhen a internaltransaction withlarge number ofrecords is com-mitted toNVRAM



NimbleOS 5.1.1.0 Resolved Critical Issues




In NimbleOS 4.5.0.0 and5.0.x, arrays with 80GBSSDs may experiencehigher read and write laten-cy caused by a lower cachehit rate. This may occurduring increased garbagecollection activity.

Lower cache hitratio on systemswith 80GB SSDs


Not applicableAfter assigning and unas-signing a member array toand from the default poolmultiple times, there couldbe stale entry in the volfamspool membership, which willfail the future pool add oper-ation and the Data Servicemay restart unexpectedlyand repeatedly.

Data Servicemay experiencerestart loop afterrepeatedly as-signing andunassigningmember arrayfrom default pool


Not applicableIf a Windows cluster hassynchronous replication en-abled on clustered volumes,it may experience IO time-outs or long latency onthose volumes after an Au-tomatic Failover (AFO) orAutomatic Switchover(ASO) is performed, whichis triggered by Group Lead-er (GL) or Backup GroupLeader (BGL) array down.

Windows clustermay experienceIO timeout orlong latency af-ter AFO/ASOtriggered byGL/BGL down


Not applicableUnder certain conditions,the Data Service on the ar-ray may restart unexpected-ly if a volume is deletedwhile it is being moved toanother pool.

Data Servicerestarts whenvolume is delet-ed while beingmoved


Not applicableAfter assigning and unas-signing a member array toand from the default poolmultiple times, the DataService may restart unex-pectedly and repeatedly.

Data Servicerestart after re-peatedly assign-ing and unas-signing memberarray from de-fault pool






If this issue occurs, makethe affected controllerstandby and power cycle it.

The SCSI High Availabilityservice may restart follow-ing controller reboot whena Fibre Channel HBA is un-recognized. This problem isrelated to a limitation in theFC HBA firmware.

SCSI High Avail-ability servicemay restart fol-lowing controllerreboot whenHBA is unrecog-nized

PlatformAS-84550

Not applicableIn rare instances, an arraythat has Netlist NVRAMcards may experience anunexpected Data Servicerestart when updating toNimbleOS 5.0.x. Thisrestart occurs due to amemory allocation issue forthese systems that is mostlikely to transpire duringsoftware updates. Duringthe software update, thekernel memory require-ments increase, and thekernel attempts to allocatefree pages for the NVRAMdriver. If this process sur-passes the configuredtimeout , it may cause theData Service to restart.

Data Servicerestart whenNetlist NVRAMarrays update toNimbleOS 5.0.x

PlatformAS-82981

Not applicableThe Fibre Channel com-mand state machine doesnot always properly handlethe situation where the DataService takes an unusuallylong time to process events.This could lead to unexpect-ed Data Service servicerestarts and session fail-ures.

Long delays inprocessing FibreChannel re-quests can leadto restarts of theData Service

PlatformAS-92877

Not applicableFollowing a NimbleOS up-date to 5.0.5.0 or 5.0.5.200,Intel SSDs running specificfirmware versions maytimeout or fail after SMARTdata is collected.

Intel SSDs maytimeout or fail af-ter update to5.0.5.0 or5.0.5.200

PlatformAS-91750





Not applicableThe NimbleOS 5.0.x.x re-lease introduced a racecondition in the iSCSI con-nection termination logic,which causes connectionprocessing logic to en-counter unexpected connec-tion state, resulting in asegmentation fault. TheData Services Daemonrestarts, and the array re-sumes servicing requestsautomatically.

Data Servicerestarts due torace conditionassociated withiSCSI connec-tion termination

SANAS-84365

Contact HPE Nimble Stor-age Support to assessworkarounds.

Prior to attempting groupmerge of two independentgroups, one group mayhave been configured withActive Directory and had anActive Directory user log in-to that group. On the othergroup, a local user with thesame name may have beencreated and logged in. Thislocal user is deleted laterbut the array keeps historyof its login. Later, if you tryto merge these two groups,the duplication of the ActiveDirectory and local username login history willcause the group merge tofail with an unexpectedGroup Management servicerestart. There may be varia-tions in this user name dupli-cation on each array groupprior to group merge.

Group Manage-ment servicemay restart unex-pectedly duringgroup merge

System Manage-ment

AS-80883





Contact HPE Nimble Stor-age Support to assessworkarounds.

When variations exist withan Active Directory loginusername on a source andtarget group involved in agroup merge operation, un-expected Group Manage-ment service restarts mayoccur during each groupmerge attempt. As a result,each group merge opera-tion will fail. For example,domain\username loggedinto the source group as theuser and just the usernamelogged into the target groupas the user or vice-versa.

Group Manage-ment servicemay restart unex-pectedly duringgroup merge

System Manage-ment

AS-80367

No workaround is available.Network configuration er-rors during group mergemay cause encryption keysto go missing from sourcegroup after the groupmerge.

Network configu-ration errors dur-ing group mergemay cause en-cryption keys togo missing

System Manage-ment

AS-84861

Not applicableIn rare cases, during certifi-cation regeneration, a racecondition can occur be-tween the array group certifi-cate regeneration processand an internal REST han-dler. As a result, the GroupManagement service mayrestart.

Group Manage-ment servicemay restart dur-ing certificate re-generation

System Manage-ment

AS-64959

Contact Nimble StorageSupport.

If the array has been updat-ed from 2.1.x.x or earlierversions in its history wheresnapshots were taken, exe-cuting replication workflowslater, such as promoting avolume collection, may leadto a series of unexpectedGroup Management servicerestarts due to a problemwith the underlying branch-es.

Group Manage-ment servicemay restart unex-pectedly duringreplication work-flows

System Manage-ment

AS-84112





Not applicableAfter successfully creatingsnapshot collections, theGroup Leader may loseconnection to the databaseand as well as the BackupGroup Leader. As a result,the Array ManagementService may restart unex-pectedly to restore the con-nections. This may becaused by network connec-tivity issues.

Array Manage-ment ServiceRestarts afterlosing connec-tion to BackupGroup Leader

System Manage-ment

AS-95207

Reduce the frequency ofapplications opening andclosing many different vol-umes’ stats.

When external applications,external scripts, multiplemonitoring GUI windows ortabs rapidly open and closeper-volume statistics acrossdifferent volumes manytimes per-second, theGroup Management servicemay restart unexpectedly.

Group Manage-ment servicemay restart unex-pectedly whenfrequently open-ing and closingstats from manydifferent vol-umes

System Manage-ment

AS-22391

Resolved Issues

Resolved Issues in NimbleOS version 5.1.1.0


Not applicableA race condition betweengarbage collection and theindex validating the livenessof blocks can cause DataService to restart becauseof an internal check. Thecondition clears itself auto-matically on a Data Servicerestart.

A race conditionbetweengarbage collec-tion and indexvalidating theliveness ofblocks cancause the DataService torestart



NimbleOS 5.1.1.0 Resolved Issues



If the GDD service restartsrepeatedly, consider thefollowing workarounds: 1)Disable re-balance, netcon-fig --edit active --iscsi_con-nection_rebalancing false2) Contact HPE NimbleStorage Support to increasethe GDD health-check to600 seconds.

The iSCSI re-balance oper-ation goes through eachconnection for each targetto evaluate if the connectioncan be moved to betterplace. In each evaluation, alookup to an internaldatabase occurs to getconfiguration parameters forthe IP and host. In configu-rations with a large numberof volumes, this lookuptakes a longer than usual toprocess all the targetswhich causes the total timeto exceed the health check.As a result, the Group Dataservice (GDD) restarts torecover.

Group Data ser-vice restarts dur-ing connectionrebalancing onsystems withmany volumes


Not applicableUnder rare circumstances,the Data Service on the ar-ray may restart unexpected-ly if the Deduplication Esti-mator tool is run on the ar-ray.

Data ServiceRestart whenDeduplicationEstimator Tool isrun


Not applicableThe garbage collector re-quires memory to keeptrack of the segments it iscleaning. The amount ofmemory configured in thesystem was constant acrossall array models, and forcertain models, this was in-sufficient. To fix this, a cal-culation is now used to pro-vision the appropriatememory based on the arraymodel.

Improvements ingarbage collec-tion memoryprovisioning


Not applicableThe Mirroring Service onthe array may restart unex-pectedly as space accumu-lates on an internal systempartition.

Mirroring Ser-vice restarts un-expectedly






Not applicableWhen the Data Service isbeing restarted or shut-down, it is possible for theservice to crash instead ofterminating gracefully. Thisis due to a bug in the pro-cessing of certain incomingmanagement operationsfrom other services. Thesemanagement operations donot recognize the processis shutting down, and re-main in the system whenthe service termination logicattempts to free some re-sources that these manage-ment operations require.

Data Servicecrash duringrestart or shut-down


Not applicableIn rare cases, network con-gestion will cause internaldelays in the Data Servicethat are misinterpreted asan error. The service will berestarted automatically torecover from this perceivederror. The system will re-sume normal operation afterthis restart. Protocol (iSCSI,FC) network congestion willnot trigger this failure. It isspecific to internal manage-ment and snapshot replica-tion traffic.

Data Servicemay restart dur-ing management/ replication net-work congestion


Not applicableAn internal data structurewithin the Data Servicestores the key-ids of encrypt-ed volumes. When encrypt-ed volumes are cloned andthose clones are deleted,this leads to incrementedkey-ids being generated. Incertain instances, a consid-erably large amount of key-ids (~50,000) may consumehigh memory and cause theData Service to restart unex-pectedly.

Data Servicerestarts due tolarge amount ofkey-ids of en-crypted volumes






If a DSD restart loop oc-curs, contact HPE NimbleStorage Support to stabilizethe system.

When a dedupe-capablearray has writable snap-shots (i.e., most commonlyassociated with VSS snap-shots) on a volume withdedupe currently disabled,then a user enables dedupeon the volume and writesdata to a writable snapshoton that volume, a Data Ser-vice restart loop conditioncan occur.

Data Servicerestart loop afterenabling dedupli-cation on a vol-ume withwritable snap-shots


Prior to this change, a ses-sion would be created butall SCSI commands wouldbe returned with BUSY sta-tus which would cause adata unavailable situation.The Fibre Channel sessionwould need to be terminat-ed to resolve this issue. Theeasiest way of doing this isto reboot the controller.

The array does not provideFibre Channel fabric ser-vices. If the initiator at-tempts to create a sessionto a Well Known Fabric Ad-dress, the session is nowexplicitly logged out. If theinitiator sends SCSI com-mands to the array withoutfirst sending a PRLI com-mand, the session will alsobe logged out. In each casethe initiator will re-establishthe session and normal Fi-bre Channel traffic will re-sume.

The Fibre Chan-nel target drivershall explicitly lo-gout sessions ifFCP commandsare receivedwithout a PRLIor if the sessionis to a WellKnown FabricAddress.

PlatformAS-86535

Not applicableAfter performing a softwareupdate to NimbleOS 5.0.x,lower-end platforms mayfalsely report multiple in-stances of the array beingin SOLO mode even thoughthe array is actually in HighAvailability mode (Active /Standby). This may occurwhen the nightly DNA isgenerated on the array. TheDNA creation causes a loadon the interconnect link be-tween the active and stand-by controller. And in someinstances, the link betweenthe two controllers is ex-hausted, causing a failurein communication.

False alerts ofthe array beingin solo mode af-ter nightly DNAis generated

PlatformAS-88785





Contact Nimble StorageSupport.

Stale rsh sessions, used in-ternally during controllercommunication, may causesoftware update processesor other services to hang.Failover typically clearsthese stale sessions but inrare cases failover too willhang requiring Nimble Stor-age Support intervention.

Software updateprocesses orother servicesmay hang due tostale rsh ses-sions

PlatformAS-60494

No workaround availableyet. EMD automaticallyrestarts and continues shelfmanagement. Future Nim-bleOS releases will have anappropriate fix.

Enclosure Managementservice restarts in rare cor-ner-corner case after issu-ing an Intelligent PlatformManagement Interface (IP-MI) command. The serviceautomatically restarts andcontinues shelf manage-ment.

Enclosure Man-agement servicemay restart afterissuing an IPMIcommand

HardwareAS-75123

NoneThe SCSI High AvailabilityService may restart unex-pectedly on the standbycontroller when a SCSI for-ward command returns afailure, but the memorybuffer is not cleared. Thisservice will recover success-fully after the restart. ER-ROR: scsi.data:f_SFDCMD-FWDOP_done: sc-si_sfd_cmd_fwd_op.cc:226:check failed: _sfw_resp_da-ta_buf == __null ERROR:scsi.data:f_SFDCMDFW-DOP_done: sc-si_sfd_cmd_fwd_op.cc:226:aborting:

SCSI HA Ser-vice restarts un-expectedly onstandby con-troller

SANAS-82674





No known workaround.SCSI commands are pro-cessed by both the DataService and the SCSI HighAvailability Service. Normal-ly, SCSI commands areprocessed to completion,and the SCSI response issent. But under certainconditions (Abort orLU_Reset task manage-ment, or connection loss),commands are terminatedabnormally instead of exe-cuting to completion. Dueto a race condition defect inthe abnormal terminationlogic, the Data Service orSCSI High Availability Ser-vice may generate a pro-cess core, and restart. Thisresults in a short disruptionin the Nimble storage sys-tems ability to service IOrequests from hosts. Aftera short period of disruption,the affected processrestarts and resumes servic-ing host IO requests.

Data Service orSCSI High Avail-ability ServiceRestart duringabnormal com-mand termina-tion

SANAS-88224

Fail-over on each array inthe group after performingthe volume resize on thearray and before performingthe extend operation on thehost, or contact HPE Nim-ble Storage Support to workaround the issue.

In a multi-array pool environ-ment, if hosts are connect-ed to the pool using group-scoped target (FC arrays,or VVols over iSCSI arrays),the host might fail to extendthe volume if the volume isresized on the array. Forexample, on Microsoft Win-dows, you may see the fol-lowing error when trying toextend a drive. “The drivecannot find the sector re-quested.”

Performing Vol-ume Resize inMulti-Array PoolConfigurationsmay cause Er-rors during HostExtend Opera-tions

SANAS-80470

Reducing the number of lo-gins or SSH sessions isrecommended.

The array may experiencean unexpected takeover af-ter an excessive amount oflogin attempts (thousands)occur in rapid succession.This may be caused by thirdparty monitoring tools orscanners.

Array may expe-rience unexpect-ed takeover afterexcessive loginattempts

SecurityAS-83463





Disabling the Kerberos testis recommended.

When a security scanner isused to perform a Kerberosauthentication for an SSLconnection, this may causethe SSL connection threadto hang. After a timeout, theGroup Management servicerestarts.

Security scannerusing Kerberosauthenticationfor SSL causesservice restart

SecurityAS-90015

Not applicableA deadlock can occur be-tween two managementprocesses when updatingthe replication bandwidththrottle. The ManagementProcess will restart andclear the condition. There isno disruption to data ser-vices and replication willcontinue automaticallywithout intervention.

Group manage-ment processmy restart due toa deadlock whenreplicating

System Manage-ment

AS-78436

If snapshots do not need tobe written to or read from,avoid adding an ACL to thesnapshot. If ACLs areadded to the snapshot,avoid bringing multiplesnapshots online simultane-ously.

When multiple snapshotsthat have an access controllist (ACL) configured to thesame set of hosts, are setonline simultaneously, it islikely that the same LUN IDcan be assigned to differentsnapshots.

Setting multiplesnapshots onlinesimultaneouslycan lead to aLUN ID conflict

System Manage-ment

AS-95570

Not applicableDue to a race condition withvolume and branch cre-ation, the Group Manage-ment service can crashwhen creating volumes.Occurrence of this racecondition is extremely rare.The Process Managementservice automaticallyrestarts the Group Manage-ment service after the crashand it should resolve therace condition.

Group Manage-ment servicemay restart dueto race condition

System Manage-ment

AS-89298

Not applicableAfter deleting the existingmasterkey and activating anew one, the followingalarm persists - Encryptiondeactivated. Encrypted vol-umes cannot be accessedor created. Enter encryptionpassphrase to reactivate.

Alarm to Enterencryptionpassphrase toreactivate is notcleared

System Manage-ment

AS-77242





Please contact HPE NimbleStorage Support to adjustthe value.

The group merge operationmay fail if an internal arrayconfiguration setting is setto an unexpected value.

Group mergefails due to unex-pected array set-ting

System Manage-ment

AS-81610

Please run the followingcommand via the HPENimble Storage Array CLIto verify that the groupmerger completed success-fully: group --info Verify theMember array(s) section toconfirm that all arrays wereproperly added to thegroup.

A group merge is initiatedvia the HPE Nimble StorageArray CLI. The CLI reportsthe failing error, suggestingthat the operation has timedout: ERROR: Failed tocommunicate with systemmanagement service...However, the group mergestill completes successfully.

CLI reports atimeout butgroup merge isstill successful

System Manage-ment

AS-75560


After and AutomaticFailover, when old GroupLeader is powered on, it re-mains out of sync due tofailing to takeover as theBackup Group Leader. Thisis a relatively rare racecondition that can happenwhen the Group Manage-ment Services on the Back-up Leader is coming up andbringing its services online,and one of the services ei-ther takes too long to comeup or fails to come up.

After an Automat-ic Failover, theprevious GroupLeader fails totakeover as theBackup GroupLeader

System Manage-ment

AS-90646

Following group leader mi-gration, if controller-levelalerts are not being sent,perform controller failoveror contact Nimble StorageSupport to manually restartthe Controller ManagementService.

In rare cases, after migrat-ing the group leader role toanother array in a multi-ar-ray group, the Controllerand Array Managementservices may become outof sync. Then, controller-level alerts cannot be sentto the Event ManagementService. As a result, thesecontroller-level alerts mayfail to be sent internally orexternally to Nimble Sup-port until the ControllerManagement Service isrestarted.

Controller-levelalerts may fail orappear delayedfollowing groupleader migration

System Manage-ment

AS-72860





Not applicableIn multi-array groups, man-agement services backupis setup to serve as a back-up in case that the leaderarray fails. However, duringthe setup of the backup, itis possible that the initialseed takes longer than thepre-determined amount oftime given for the seedingto finish. If we try to repli-cate changes to thedatabase synchronouslyduring the seeding, the pro-cess become unresponsive.As a result, the Array Man-agement service restarts.

Array Manage-ment servicemay restart dueto deadlock

System Manage-ment

AS-71088

Not applicableIf netconfig validation failedduring group setup, theiSCSI target name used forvVol is not generated cor-rectly. Subsequently, whenvVol integration is enabled,the Group Managementservice will terminate unex-pectedly when querying forProtocol Endpoint at-tributes.

Group Manage-ment service ter-minates unex-pectedly whenquery for Proto-col Endpoint at-tributes

System Manage-ment

AS-74838


HPE Cloud Volumes withdeduplication enabled mayfail to replicate. This occursif the volume uses a customperformance policy, and theapplication category is notdefined.

HPE Cloud Vol-ume replicationmay fail if appli-cation categoryis not configuredfor custom per-formance poli-cies

System Manage-ment

AS-83269

Not applicableUnder certain conditions,the Group ManagementService may restart unex-pectedly on the array. Thismay occur if an excessiveamount of snapshot opera-tions (create snapshot,delete snapshot, restorevolume) are run on thesame volume, for a largeamount of volumes.

Group Manage-ment Servicemay restart dueto large amountof snapshot oper-ations

System Manage-ment

AS-90226





Not applicableThe Group ManagementService may restart unex-pectedly on the downstreamarray during the initial repli-cation configuration syncwith the upstream array.This may occur if there area large amount of volumesbelonging to the volumecollection being replicatedand if the volumes have alarge amount of FibreChannel (FC) connections.

Group Manage-ment Servicerestarts unex-pectedly ondownstream ar-ray due to exces-sive FC ses-sions

System Manage-ment

AS-90913

Not applicableDue to a rare timing issue,the Group Managementservice may restart unex-pectedly if a snapshot collec-tion is being deleted whilean internal checkpoint isbeing performed on thesame collection.

Group Manage-ment ServiceRestarts whensnapshot collec-tion is deleted

System Manage-ment

AS-90569

Not applicableUnder rare circumstances,the Group Managementservice on the array mayrestart unexpectedly after areplication partner is delet-ed. This occurs if a periodicconnection task betweenthe partners happens to berunning at the same time.

Group Manage-ment ServiceRestarts afterdeleting replica-tion partner

System Manage-ment

AS-91868

Not applicablePerforming a dissociation ofa volume from a volumecollection while replicationis in progress can, in rarecases, lead to a restart ofthe Group Managementservices. This is not criticalas the group managementservices are restarted auto-matically and replicationcontinues from where it leftoff.

Volume dissoca-tion during repli-cation maycause GroupManagementservice restart

System Manage-ment

AS-87008

Not applicableDue to a timing issue, theGroup Management Servicemay restart unexpectedlywhen it is querying a snap-shot that has already beendeleted.

Group Manage-ment ServiceRestarts due totiming issue

System Manage-ment

AS-87921





Not applicableDue to a race conditionwhen deleting empty snap-shot collections, the GroupManagement service cancrash unexpectedly. Thisissue is resolved when theGroup Management serviceautomatically restarts. TheProcess Management ser-vice automatically restartsthe Group Managementservice after the crash andit should resolve the racecondition.

Group Manage-ment servicemay restart dueto race conditionwith snapshotcollections

System Manage-ment

AS-90407

Not applicableDue to a rare race condi-tion, the Group Manage-ment Service may attemptto resume replication for asnapshot collection whosereplication was abortedearlier. This will cause theservice to restart unexpect-edly.

Group Manage-ment Servicerestarts unex-pectedly

System Manage-ment

AS-84360

Prior to deleting volumecollection, delete snapshotcollections associated tovolume collection if snap-shot collections are not re-quired/desired. Even thoughprotection schedule isdeleted, snapshot collec-tions are accessible andcan be deleted from CLI orGUI, if desired.

Deleting volume collectionor deleting protectionschedule from a volumecollection will leave behindsnapshots that will not beautomatically deleted. Suchsnapshots can occupyspace.

Deleting volumecollection ordeleting protec-tion schedulecan leave be-hind snapshots

System Manage-ment

AS-18597

Modify the retention policysuch that a fraction of thetotal number of snapshotsto be discarded due to thepolicy change are sched-uled for deletion, then waituntil the number of snap-shots equals the new value.Repeat this step until all re-tention policies have thedesired value.

The manual reduction ofscheduled snapshot reten-tion policies for any numberof volume collections mayresult in multiple systemmanagement servicerestarts when the reductionresults in a large number ofsnapshot deletions. Systemmanagement service willstabilize once the numberof snapshots equals thenew retention value acrossall volume collections.

System manage-ment servicemay restartwhile reducingsnapshot reten-tion for volcolls

System Manage-ment

AS-77649





Not applicableWhen accessing a specificinternal reference variablethat has been deleted, theArray Management servicemay unexpectedly restart.This condition is more likelyto occur with very high ratesof reads of multiple stats.Third-party monitoring appli-cations are known to ac-cess all per-volume statis-tics in this manner. The Ar-ray Management servicerestart automatically recov-ers the condition.

Array Manage-ment servicemay restartwhen readingmany differentstatistics at ahigh rate

System Manage-ment

AS-81890

Minimize opening statsmany times with an automat-ed tool or script. If the ser-vice remains unstable, dis-continue usage of customor third party stats monitor-ing applications.

The Array Managementservice can restart due toleaking memory. This oc-curs when a client repetitive-ly opens stats which do notexist. The stat open re-quests are usually per-formed by stats CLI, andare usually performed by a3rd party automated statsdata collection tool. Dataservices are not adverselyaffected by the Array Man-agement service restart.

Array Manage-ment serviceleaks memoryduring somestats read ofnon-existentstats data

System Manage-ment

AS-83040

Not applicableDue to a memory allocationissue, the Active DirectoryIntegration Service on thearray may restart unexpect-edly, causing Active Directo-ry RBAC authentication tofail.

Active DirectoryIntegration Ser-vice fails restartsunexpectedly

System Manage-ment

AS-73687

Not applicableIn rare instances, the GroupManagement Service mayrestart while an Active Direc-tory user attempts to log in.This occurs if the arraysKerberos ticket with the ADserver expires and is stucktrying to renegotiate. TheGroup Management Servicerestarts as part of a routinehealth check if the renegoti-ation surpasses five min-utes.

Stale Kerberosticket with ADmay causeGroup Manage-ment servicerestart

System Manage-ment

AS-78805





Not applicableDue to a section of codethat is not thread safe, con-current user logins cancause the Group Manage-ment service to restart unex-pectedly.

The Group Man-agement servicemay restart dur-ing concurrentuser logins.

System Manage-ment

AS-82827

Not applicableIn certain circumstances,due to a rare race condition,the Group Managementservice may restart unex-pectedly on the array. Thismay occur if a login is madeto the array while internalsystem log maintenance isbring performed.

Group Manage-ment Servicemay restartwhen logging in-to array

System Manage-ment

AS-82976

Not applicableAs part of the 5.1.1.0 Re-lease, HPE Nimble StorageRelease now supports vol-umes with the same namewithin a group as long asthey are in different pools.Previously, the volumename was unique in thegroup, but now they are on-ly unique within a pool. Tomake identifying the correctvolume easier, we now in-clude the pool name in theevent/alert. Tools that parsethe events/alerts and arelooking for the volumename should take into ac-count the additional informa-tion in the event/alert

Events andAlarms for vol-umes now in-clude pool name

System Manage-ment

AS-75227

If you see the dialog box in-dicating the error, pleasewait for 60 seconds and re-fresh the HPE Nimble Stor-age Array GUI. You mayalso need to clear thebrowser cache if the issuepersists.

A dialog box indicating thatThe web service is unreach-able or very slow may ap-pear within the HPE NimbleStorage Array GUI due tointermittent communicationproblems, even when thereis no user activity in theGUI. This is caused by auto-matic data refresh requestsoccurring in the back-ground.

The web serviceis unreachableor very slow isreported withinthe array GUI

System Manage-ment

AS-69937





Ensure that there is not aLUN ID conflict between thetwo groups.

When attempting to mergetwo Fibre Channel groups,if there is a conflict in theLUN IDs, the merge will failwith the wrong error mes-sage. The incorrect errormessage states conflictingalias name when LUN IDsactually have a conflict.

During a groupmerge of two Fi-bre Channelgroups, a LUNID conflict isidentified with anincorrect errormessage

System Manage-ment

AS-66521

Known Issues

Known Issues in NimbleOS version 5.1.1.0


Not applicableDuring a planned shutdownof an array that has syn-chronous replication down-stream volumes, a longrunning integrity check cancause the Data Service ofthe array to restart unexpect-edly. The latency in the in-tegrity check can be intro-duced due to slownesswhile syncing data / metada-ta to disk.

A long runningintegrity checkon sync repldownstream vol-umes causesData Servicerestart duringshutdown


Not applicableIn certain scenarios, slowresync progression on largevolumes using SynchronousReplication may delay thetransition of other volumes- that have already finishedtheir resync - from out-of-sync state to the in-syncstate. All volumes shouldeventually reach the in-syncstate.

Initial resync oflarge volumesmay delay othervolumes transi-tions from out-of-sync to in-sync.


Retry operation after a fewminutes to reassign array topool.

Assigning an array to a poolimmediately after unassign-ing it from the same poolwill fail with the followingerror - Failed to assign ar-rays to the pool: A serviceis not running or is notreachable

Unassigning andreassigning ar-ray to a poolwithin 5 minuteswill fail



NimbleOS 5.1.1.0 Known Issues



To see the correct poolswithin the group, close thecreateDatastore dialog andlaunch the dialog again, se-lecting the group for creat-ing datastore.

When attempting to createa VVol datastore, changingthe selection of the groupdoes not fetch the latestgroup-related pools data.This is caused by a cachingissue.

Error Could notfind pool in VVoldatastore cre-ation wizard

Host IntegrationAS-92898

Not applicablevCenter Server LinkedMode allows the user to lo-gin to any one vCenterserver and view and man-age the inventories of all thevCenter server systems ingroup. The vCenter PluginHTML5 Client is not support-ed to work with a vCenterserver in the Linked Modeconfiguration.

vCenter PluginHTML5 Clientdoes not supportvCenter ServerLinked Mode

Host IntegrationAS-90717

In rare instances due toslow IO to one of the disks,the Data Service may hit atimeout and restart unex-pectedly. This issue mayoccur during a NimbleOSsoftware update.

Data Servicerestarts due toslow IO to one ofthe disks

PlatformAS-90096

Contact HPE Nimble Stor-age Support.

When the SAS HBA detectsfaulty states, to recover, thearray needs to reset theSAS HBA’s firmware. TheSAS HBA firmware resetcan block disk I/Os signifi-cantly longer than our HAmonitoring timeouts allow.Instead, a controller rebootis triggered immediately ifthis state is detected, result-ing in an unexpectedtakeover event.

Unexpectedcontrollertakeover due toincorrect state ofthe SAS HBA

PlatformAS-33725





Active controller will be fineafter the faulty standbycontroller is disabled. (1)Pull out the rebootingstandby controller, or (2)Contact HPE Nimble Stor-age Support to remotelypower off the rebootingstandby controller from thearrays active controller.Then reseat or disk --re-move | add the failed SSDsto recover the SSDs.

In a rare case, when thestandby controller is in areboot loop, this may dis-rupt the Data Service. As aresult, the array will reportseveral failed SSDs.

Data Servicedisrupted whenstandby con-troller is in a re-boot loop

PlatformAS-49747

Address the network storm,and then perform a con-troller failover.

On lower-end platforms witha lower number of CPUcores, a network stormcould keep the kernel busyserving network interruptrequests, and ultimatelydelay IO requests. This inturn causes IOs to timeoutand cause a Data Servicerestart to recover.

Network stormcauses writes toslow down to apoint where Da-ta Servicerestarts due to ahealth check fail-ure

PlatformAS-55765


In rare cases, heavy loadcompounded by unalignedworkloads on high-endplatforms with many CPUcores can cause theNVRAM driver to becomeoverloaded when all coresare busy. The Data Servicemay restart unexpectedlywith a health check error.

NVRAM over-loaded on highend platformsunder heavystress

PlatformAS-56600


In rare cases, multi-bit ECCerrors on the NVRAM cardmay cause the Data Serviceto restart unexpectedly.

Multi-bit Error-Correcting Code(ECC) errors onNVRAM cardcaused DataService restart

PlatformAS-56942

Schedule time for a SalesEngineer to field-image thearray.

Upon removing an arrayfrom a group and removingany expansion shelves fromthat array, the array willneed to be re-imaged priorto any array reuse.

Removing an ar-ray from a groupleaves the arrayin an unusablestate.

PlatformAS-32895





Offline the read-only snap-shots where the hosts issending write requests.

When host sends write re-quests to read-only snap-shots, the writes will fail withSCSI Check Condition withadditional status SC-SI_ASC_ASCQ_LU_SOFT-WARE_WRITE_PROTECT-ED. But if Volume Manageris creating snapshot at thesame time, the Data Ser-vice may hit an assert andrestart to recover automati-cally.

Data Servicemay hit assertwhen host sendswrites to read-only snapshotand snapshotcreation is inprogress.

SANAS-86843

There is no workaround.In NimbleOS, the Data Ser-vice and the SCSI HighAvailability Service use thesame logic for processingFibre Channel connections.Due to a software defect inthe Fibre Channel connec-tion termination logic, eitherof these services may stopunexpectedly with a corefileand restart. The softwaredefect occurs when code-paths related to connectiontermination are simultane-ously processed: - process-ing a Connection_Lossevent from the FC driver, forconnection A - performingan implicit logout of connec-tion A, due to receipt of aNew_Connection event fora conflicting connection B(between the same host/ar-ray ports). The restart of theaffected service causes abrief interruption in the ar-rays ability to service incom-ing requests.

Data Service orSCSI High Avail-ability ServiceRestart whenprocessing FibreChannel connec-tions

SANAS-94539





After volume migration hascompleted, the NimbleOSData Service disconnectsany iSCSI connection estab-lished from a host, to thevolume-scoped iSCSI targeton the source array. Thehost will attempt to reestab-lish the iSCSI connection,using either the arrays Dis-covery IP Address (as re-ported during iSCSI discov-ery), OR directly to the DataIP Address on the sourcearray. If the host chooses toreconnect to the Data IPAddress, NimbleOS re-sponds with an iSCSI TEM-PORARY_REDIRECT re-sponse, to one of the DataIPs on the destination array.But due to a defect in theNimbleOS logic that pro-cesses such misdirect-ed/stale iSCSI login at-tempts, if hosts repeatedlyperform such misdirectedlogins to stale Data IPs, theData Service will eventuallystop accepting any newiSCSI connections.Restarting the Data Servicewill allow NimbleOS to re-sume accepting new iSCSIconnections.

Data Servicestops acceptingnew iSCSI con-nections aftervolume migrate

SANAS-95296

None required. The processis shutting down.

While the group manage-ment process is shuttingdown, it may experience afault that causes a restartattempt.

Group Manage-ment restartswhile shuttingdown

SecurityAS-56019

Validate the DNS serversdefined on the array andeliminate any inaccessibleor unresponsive DNSservers. If all defined DNSservers have been con-firmed and the Event ser-vice continues to crash,contact HPE Nimble Stor-age Support.

Due to the use ofSIGALARM signals to indi-cate a DNS timeout inlibcurls standard system re-solver, when the signalhappens it can sometimesswitch the application con-text to the signal handler,causing the Event serviceto crash unexpectedly.

The Event ser-vice may crashduring applica-tion contextswitch

System Manage-ment

AS-82317





Correct the DNS or SMTPconfiguration to a valid ad-dress by ensuring that aping to the defined addresssucceeds.

We create multiple threadsto deliver emails, but weuse a non-threadsafe libcurlcall to dispatch them.Therefore, the lock needsto be around libcurl call. Ifthere is a misconfiguredDNS or SMTP server, thecurl call will timeout. If thereare greater than 7 emailswaiting to be delivered andall are suffering a timeout,we will starve the healthchecking for more than the300-second health checktimeout causing the EventManagement service torestart.

Flood of time-outs causingEvent Manage-ment servicerestart

System Manage-ment

AS-68651

Re-run array removal toclear the group leadermetadata.

In some cases when thereis high load, array removalcan take longer than usualand timeout, which leads toan intermediate state wherethe group leader believesthe array still belongs to thegroup, but the array hasbeen removed.

Array removalmight not com-plete in time

System Manage-ment

AS-28992

Not applicableIf a user tries to create anew user account, but theuser doesnt have the privi-lege to do so, the user cre-ation will fail. However, anaudit log entry is not creat-ed.

No Audit Log en-try is created ifuser does nothave the privi-lege to createuser

System Manage-ment

AS-71090

Not applicableIn rare cases, a memory in-consistency causes an un-expected Group Manage-ment Daemon (GMD)restart.

UnexpectedGroup Manage-ment restart dueto memory incon-sistency

System Manage-ment

AS-55005





Restarting the AutomaticSwitchover service on theBackup Group Leader recov-ers from this failure. Thiscan be done by issuing afailover on the BackupGroup Leader.

In rare instances, if thegroup --migrate commandis issued on the GroupLeader array and the Back-up Group Leaders Automat-ic Switchover Service hap-pens to restart at the sametime (due to different rea-son), it is possible that theBackup Group Leader wontbe able to promote itself asleader.

Issuing group --migrate could re-sult with noGroup Leader ifthere is a failureon Backup lead-er

System Manage-ment

AS-95263

No workaround is needed.The service recovers on itsown.

In Automatic Switchoverenvironments, in rare in-stances, network isolationof the Group Leader andBackup Group Leader maycause the service that han-dles the automaticswitchovers to restart unex-pectedly.

Network isola-tion of the GroupLeader andBackup GroupLeader arraymay lead to Auto-maticSwitchover ser-vice restarts

System Manage-ment

AS-94683

Not applicableThere is a --force switchavailable when deleting aperformance policy via theHPE Nimble Storage ArrayCLI. This --force switchdoes not work and will failwith the following error:ERROR: Failed to deleteperformance policy. Re-source busy. The --forcecommand is not supportedsince the specified perfor-mance policy should not beremoved without firstchecking its volume or fold-er associations.

Force deletion ofuser definedperfromancepolicy should notbe supported

System Manage-ment

AS-74242

Not applicableIn rare instances, the ArrayManagement Service mayrestart unexpectedly whilethe Backup Group Leaderis being setup or removed.This restart is non-disrup-tive to the data on the array,and the Array Managementrecovers after the restartoccurs.

Array Manage-ment Servicerestarts duringBackup GroupLeader Setup orRemoval

System Manage-ment

AS-94836





Please reissue the com-mand. If the operation wasalready performed by theearlier command, an appro-priate message will be re-turned.

Under system busy condi-tions, when an excessiveamount of operations arebeing issued in parallel ortoo many internal retries areoccurring to perform tasks,you may receive a No mes-sage received error after is-suing a CLI command.

Error No mes-sage receivedafter issuing CLIcommand

System Manage-ment

AS-95429

Not applicableThere is currently no way tolist volumes using Syn-chronous Replication viathe array CLI.

Unable to listvolumes usingSynchronousReplication viaCLI

System Manage-ment

AS-92380

Not applicableThe group --list_limits CLIcommand does not list theSynchronous Replicationvolume count Limit. Syn-chronous Replication on5.1.0.0 and later can protectup to 128 volumes.

SynchronousReplication Vol-ume Count Limit

System Manage-ment

AS-89124

Not applicableOn rare occasions, a veryslow network (or networkissues like dropped pack-ets) can cause a restart ofthe Group Managementservices. However, replica-tion and other services willcontinue from where theyleft off prior to the restart.

Network issuesmay causerestart of GroupManagementservice

System Manage-ment

AS-28589

Not applicableWhen running the partner --list command via the arrayCLI, the output shows alarge throttle limit set for thereplication partner. This in-formation is inaccurate.

Replication part-ner throttle limitis displayed in-correctly

System Manage-ment

AS-89950





Not applicableAdding or removing vol-umes to a volume collectionwhile replication is inprogress for that volumecollection could potentiallylead to a restart of theGroup Management ser-vices. This does not have asignificant impact however,since replication (and otherservices) continue fromwhere they left off beforethe restart

Adding/removingvolumes whilereplication is inprogress maycause GroupManagementrestart

System Manage-ment

AS-44941

If a user needs to delete areplica volume, the replicavolume must first beclaimed (i.e., change owner-ship to the current group).The CLAIM option is avail-able in the detail page forthe replica volume.

A replica volume cannot bedeleted, which is why thedelete operation is notavailable within the HPENimble Storage Array GUI.

Replica volumecannot be delet-ed

System Manage-ment

AS-71602

Set downstream partner onall schedules in the volumecollection to none andthereafter change replica-tion partner on desiredschedules to the new part-ner.

A volume collection cancurrently replicate to onlyone replication partner. Incase of multiple schedulesin a volume collection, edit-ing single protection sched-ule to different replicationpartner violates this con-straint.

Changing replica-tion partner in avolume collec-tion with multipleschedulesthrows error

System Manage-ment

AS-23891

Not applicableIf clones are created usingan unmanaged snapshot,then this unmanaged snap-shot will not be deletedeven if cleanup is enabled.

Unmanagedsnapshots re-main aftercleanup is en-abled

System Manage-ment

AS-93113

Not applicableIn rare cases, the standardmemory allocator does notreuse freed memory effi-ciently during heavy work-loads. When Group Manage-ment Daemon (GMD) utiliza-tion becomes too high,GMD may restart to recov-er.

UnexpectedGroup Manage-ment restart dueto high memoryutilization

System Manage-ment

AS-49590





Not applicableThe usersession list com-mand may show an expira-tion time that is invalid. Thisoccurs since it takes 5 min-utes for the in-memory expi-ration time for each user tosync.

usersession listmay shows in-valid expirationtime

System Manage-ment

AS-95221

Confirm all Domain Con-trollers in the Active Directo-ry environment that is inte-grated with the array arereachable.

In environments with an ar-ray integrated with ActiveDirectory, the create/vali-date session code holds alock on an underlying ses-sions table and then goesto Active Directory to collectmore information about theuser. If this operation takestoo long, the Group Manage-ment service may timeoutto free this lock.

Group Manage-ment servicemay restartwhile collectinguser informationfrom Active Di-rectory

System Manage-ment

AS-85951

Once the array is able todetermine its deduplicationcapability, all newly createdvolumes will have dedupeenabled, if specified. In or-der to enable dedupe on thepreviously created volumes,you may run the followingcommand via the HPENimble Storage Array CLI:vol --edit <vol_name>--dedupe_enabled yes

After a CSx000 array is in-stalled, it takes one minutefor the array to determineits deduplication capability.If a volume is created priorto this, it will not havededupe enabled even if thearray is dedupe capable.

Unable to creatededupe enabledvolumes on anew install

System Manage-ment

AS-86545

Not applicableCurrently, there is no CLIsupport for changing thewitness port. The nimble-witnessd.service file needsto be edited manually.

NO CLI supportfor changing theWitness Port

System Manage-ment

AS-92157

Delete the appropriate pro-tection schedule with asnapshot interval of lessthan five minutes to meetthe system limits of theseprotections schedules, andretry the group merge oper-ation.

The GUI shows a generalerror message during groupmerge when the total countof snapshot schedules witha snapshot interval of lessthan five minutes exceedsfive: System limits for thenumber of protectionschedules would be violatedafter adding the array. Cur-rent limit is 5.

GUI shows ageneral errormessage duringgroup mergewhen, for thetwo groups, thetotal number ofsnapshot sched-ules with a snap-shot interval ofless than fiveminutes exceedsfive

System Manage-ment

AS-21697





To work around this issue:1. Log into the destinationarray to resolve the con-flicts. 2. Attempt the groupmerge again.

Currently within the HPENimble Storage Array GUI,when performing a groupmerge, if there is a largeamount of group mergeconflicts (1000 or more), theGUI is unable to processand resolve all of them.

Group Merge viaGUI unable toprocess largeamount of con-flicts

System Manage-ment

AS-77372

No workaround. When userperforms a pool merge andhe sees the error that poolmerge cannot happen sincewitness is configured, thenif user has sync replicationsetup, do not remove thewitness hoping pool mergewill succeed after removal.

Pool merge is not allowedif sync replication is en-abled. Also pool merge isnot allowed when there is awitness configured. Herewe have a system that hasa witness configured forgroup leader failover andhas sync replication config-ured. When user tries to doa pool merge we throw anerror saying “pool merge isnot allowed when witnessis configured”. So in thiscase user removed the wit-ness and then tried poolmerge again hoping it towork. Now he gets an errorsaying “Pool merge is notallowed when involved insync replication”. So theuser unnecessarily removedthe witness which risksfailover which is needed forsync replication.

incorrect order-ing of error mes-sages

System Manage-ment

AS-95591

Run the following commandvia CLI: snap --list --all --unmanaged

The Array GUI does notspecify which snapshots areunmanaged and no longerbelong to a volume collec-tion.

Array GUI doesnot specifywhich snapshotsare unmanaged

System Manage-ment

AS-93157

Use volume collections tocheck syncRep volumes, oruse other filters to meet theneeds

There is currently no way tofilter volumes using Syn-chronous Replication withinthe array GUI.

Unable to Filtervolumes usingSynchronousReplication

System Manage-ment

AS-92379



NimbleOS 5.1.1.0 Release Notes - Hewlett Packard Enterprise...NimbleOS 5.1.1.0 Version: 5.1.1.0...

Documents

Transcript of NimbleOS 5.1.1.0 Release Notes - Hewlett Packard Enterprise...NimbleOS 5.1.1.0 Version: 5.1.1.0...