HPE 3PAR Priority Optimization · HPE 3PAR Priority Optimization software for H PE 3PAR StoreServ...

HPE 3PAR Priority Optimization Agile quality of service management on HPE 3PAR StoreServ Storage

Technical white paper


Contents Executive summary ............................................................................................................................................................................................................................................................................ 3 Terminology.............................................................................................................................................................................................................................................................................................. 4 Requirements .......................................................................................................................................................................................................................................................................................... 4 Use cases ..................................................................................................................................................................................................................................................................................................... 4

Prioritize production workloads over testing and development ........................................................................................................................................................ 4 Accommodate bursts in workloads ............................................................................................................................................................................................................................... 5 Protect from rogue processes............................................................................................................................................................................................................................................ 5 Manage application performance expectation levels ................................................................................................................................................................................... 5 Manage periodic variations in workloads ................................................................................................................................................................................................................ 5 Improving the multitenant and cloud experience ........................................................................................................................................................................................... 5 Federated QoS with HPE 3PAR Peer Motion software .............................................................................................................................................................................. 5

Virtual volume sets and virtual domains ........................................................................................................................................................................................................................ 5 Concepts and features .................................................................................................................................................................................................................................................................... 6

Mode of operation ........................................................................................................................................................................................................................................................................ 7 QoS rule attributes ....................................................................................................................................................................................................................................................................... 9 The “all_others” System rule ................................................................................................................................................................................................................................................ 9 QoS rule actions .............................................................................................................................................................................................................................................................................. 9

Overlapping QoS rules ................................................................................................................................................................................................................................................................. 10 QoS on virtual and physical copies ................................................................................................................................................................................................................................... 10 Managing Priority Optimization .......................................................................................................................................................................................................................................... 11

Using the HPE 3PAR StoreServ Management Console ......................................................................................................................................................................... 11 Using the HPE 3PAR Command Line Interface ............................................................................................................................................................................................. 15 Using Web services API ........................................................................................................................................................................................................................................................ 16

HPE 3PAR StoreServ software interoperability .................................................................................................................................................................................................... 16 Application interoperability ..................................................................................................................................................................................................................................................... 16

Databases .......................................................................................................................................................................................................................................................................................... 16 Mail systems .................................................................................................................................................................................................................................................................................... 17 Virtualization software ........................................................................................................................................................................................................................................................... 17 OpenStack ......................................................................................................................................................................................................................................................................................... 18

Reporting .................................................................................................................................................................................................................................................................................................. 18 Event and alert management ................................................................................................................................................................................................................................................ 21


Best practices ....................................................................................................................................................................................................................................................................................... 22 Assembling VVs into VVsets and virtual domains ....................................................................................................................................................................................... 22 Determining the IOPS and bandwidth capacity for a system ............................................................................................................................................................ 22 Setting up QoS rules ................................................................................................................................................................................................................................................................ 22 Implementing the System rule ....................................................................................................................................................................................................................................... 23 Handling Tier 1 applications ............................................................................................................................................................................................................................................ 23 Setting correct latency goals............................................................................................................................................................................................................................................ 23 Setting Min Goals ....................................................................................................................................................................................................................................................................... 24 Using alert criteria to monitor QoS-related metrics .................................................................................................................................................................................... 24 Using the latency goal to accommodate bursts in workloads ........................................................................................................................................................... 24 Overprovisioning performance limits ....................................................................................................................................................................................................................... 26 Priority Optimization with virtual domains .......................................................................................................................................................................................................... 26 QoS and Remote Copy .......................................................................................................................................................................................................................................................... 27 Influence of QoS on the host side................................................................................................................................................................................................................................ 27 Maximum number of QoS rules per VV ................................................................................................................................................................................................................. 27 QoS on a subset of VVset volumes ............................................................................................................................................................................................................................ 28 QoS and snapshots ................................................................................................................................................................................................................................................................... 28

Conclusion ............................................................................................................................................................................................................................................................................................... 28

Technical white paper Page 3

Executive summary Over the last decade, the storage industry has undergone a trend toward consolidations. Enterprise organizations have moved their locally attached storage to large-scale SAN-based storage systems that can hold thousands of individual disk drives. Thanks to this surge in consolidation, it is common to see hundreds of physical and virtual servers, each running one or more workloads connected to a single storage system. Consolidation has reduced the complexity of data storage and helps make management, occupied floor space, and energy consumption more efficient. It has also successfully lowered the total cost of ownership for storage in organizations.

As an immediate consequence of consolidating data from many possibly dissimilar workloads onto a single storage system, contention occurs for shared resources on the storage system. Examples of such shared resources include front-end host Fibre Channel, iSCSI, and Fibre Channel over Ethernet (FCoE) ports and adapters, back-end Fibre Channel and serial-attached SCSI (SAS) disk paths, physical disks, data and control cache, ASICs, processors, and backplane interconnections. I/O requests arriving at the front-end ports are handled on a first-come, first-serve basis. On the surface, this might sound fair, but it is unlikely to provide equal or consistent throughput for multiple concurrent workloads. By applying a priority policy to the incoming I/O requests, you can maintain the quality of service (QoS) and performance level for a workload when competition occurs for the array’s shared resources.

HPE 3PAR Priority Optimization software for HPE 3PAR StoreServ Storage systems implements and manages a priority policy per virtual volume set (VVset) that serves as a proxy for applications and per virtual domains that serves as a proxy for customers or tenants. Volumes that are not a member of a VVset or a virtual domain can be controlled using the System QoS rule. Priority Optimization enables users to take full control of performance by specifying minimum and maximum limits (“Min Goal” and “Max Limit”) for IOPS and bandwidth, as well as a priority for every QoS object along with the ability to define latency goals for the most important applications. If these goals are not met, the system automatically adjusts the service levels of lower-priority applications and workloads in order to make sure that necessary QoS levels for the highest priority applications are always maintained. This paradigm is valid on HPE 3PAR StoreServ systems because all applications on the array share all system resources including the physical disks. Applying Priority Optimization to virtual domains protects applications from monopolizing resources by a single application or single tenant. These capabilities remove the last barrier to large-scale consolidation by allowing you to deliver assured QoS levels without having to physically partition resources or maintain discrete storage silos.

Competitive QoS implementations require that you assign a workload to a predefined priority level or logically partition the array to reserve part of its resources on a semi-permanent basis to a particular workload. These solutions are neither flexible nor allow real-time enforcement. Priority Optimization is granular, versatile, and easy to configure and monitor. It requires little supervision from storage system administrators.

Priority Optimization is resident on the storage system and runs entirely on the HPE 3PAR StoreServ system. There are no host-based components to install, which means enabling the software is as simple as installing the license key for it.

At first glance, Priority Optimization appears to compete with server-side workload managers. Server-side solutions manage only I/O resources for workloads confined to the local server. Because storage-side QoS solutions manage workload I/O requests across multiple servers sharing a common storage array, administering workload prioritization is more efficient.

Priority Optimization was introduced as an integrated feature of HPE 3PAR OS 3.1.2 MU3. HPE 3PAR OS 3.1.3 added Min Goals, latency goals, and priority levels for QoS objects. These features give users full control to provision capacity and performance with confidence by leveraging HPE 3PAR leading virtualization technologies. Figure 1 shows the position of Priority Optimization in the HPE 3PAR StoreServ virtualization chain.

Figure 1. HPE 3PAR StoreServ virtualization enables you to provision capacity and performance with confidence.


This paper is based on HPE 3PAR OS 3.3.1 MU3, HPE 3PAR StoreServ Management Console (SSMC) 3.5, and HPE 3PAR Command Line Interface (CLI) 3.3.1 MU3. It provides readers with a thorough understanding of the concepts and operational aspects of Priority Optimization. It helps to have a good knowledge of the management of HPE 3PAR StoreServ systems when reading this paper. For more information on HPE 3PAR StoreServ systems and software, visit the Hewlett Packard Enterprise (HPE) website.

Configuring Priority Optimization can be part of a packaged contract or a custom service delivered by HPE Pointnext. These services offer expertise, best practices, and automation to deliver a successful QoS configuration. Consult your HPE representative or HPE partner for more information.

Terminology This paper uses the following metrics to characterize workloads for Priority Optimization software:

• Response time: The average time measured for how long the array takes to process a received I/O request. On an HPE 3PAR StoreServ system, this is reported as service time. Response time is measured in milliseconds.

• Latency: An equivalent to response time and service time. Latency is measured in milliseconds.

• Queue depth: The instantaneous number of I/O jobs that are processing and waiting to be processed at the end of a polling interval. The queue depth is a non-negative integer number.

• Bandwidth: A measure of the amount of data processed by the array per unit of time. It is typically measured in MB/s and is also referred to as throughput.

• IOPS: The acronym for “input/output operations per second,” which is a common performance metric that benchmarks computer storage. It is indicative of how many host I/O requests the array is processing per second and is provided as a non-negative number such as 50,000 IOPS.

Requirements Priority Optimization requires HPE 3PAR OS 3.1.2 MU2 or later. It is supported on all HPE 3PAR StoreServ systems that are certified for this version of HPE 3PAR OS and later. Any incompatibility is listed on the Single Point of Connectivity Knowledge (SPOCK) website. Volumes must be exported to hosts via Fibre Channel, iSCSI, or FCoE and configured in VVsets or as part of a virtual domain to be subject to a QoS rule or policy. Volumes that are not a member of a VVset or a virtual domain can be controlled using the System QoS rule.

A valid license for Priority Optimization is required on the HPE 3PAR StoreServ system. HPE updated its licensing policy for HPE 3PAR StoreServ 8000, 9000, and 20000 systems on February 13, 2017. An eternal Priority Optimization license is now part of the All-inclusive Single System Software bundle. This collection of software ships with each array purchased after that date. If the array was purchased before that date or if the array is an HPE 3PAR StoreServ 7000 or 10000, a separate spindle-based Priority Optimization license or a license as part of the HPE 3PAR Data Optimization Software Suites is available. These licenses can be converted for a small fee to the All-inclusive licensing model. Consult your HPE representative or HPE partner for more licensing information.

Creating and managing QoS definitions requires HPE 3PAR SSMC 2.1 or later. To use the command line, you must install HPE 3PAR CLI 3.1.2 MU2 or later or use the Web Services Application Programming Interface (WSAPI) 1.3 or later framework. Reports on Priority Optimization are available via HPE 3PAR System Reporter 3.1 MU1 or later, and on HPE 3PAR SSMC 2.1 or later.

Use cases Priority Optimization has several use cases that illustrate its business value. Some important use cases are highlighted in this section.

Prioritize production workloads over testing and development In all environments, production workloads take precedence over testing and development activities. Software performance and scale-up test scenarios might deplete array resources and impact production applications if the production workloads are running on the same storage system. Purchasing two separate arrays or partitioning a single array to isolate these workloads has long been the recommended approach. HPE 3PAR StoreServ systems accommodate multiple dissimilar workloads, thanks to their wide striping architecture and tiered storage implementation. By complementing these features with Priority Optimization, storage administrators can control and balance the distribution of array resources across multiple production and test and development workloads on a single HPE 3PAR StoreServ system. Snapshots of the production database for analysis, backup, and development purposes can be made available to the test and development organization with a lower I/O priority using QoS.

https://www.hpe.com/us/en/storage/3par.html

http://www.hpe.com/us/en/services.html

https://h20272.www2.hpe.com/SPOCK/


Accommodate bursts in workloads Most storage workloads are prone to performance bursts. Creating scientific calculation checkpoints and performing full table scans and batch updates of databases, page cache flushes, large file copies, or large-scale code compilation jobs tend to occur suddenly, requesting a sharp increase in the amount of performance from the storage system. Priority Optimization introduces a latency goal parameter per VVset, per virtual domain or for all volumes that belong to neither a VVset nor a virtual domain. Mission-critical workloads are granted IOPS and bandwidth to accommodate bursts if their defined latency goal is not met. I/O for other workloads is reduced to their minimum goals to make sure the latency of the mission-critical workloads is met. If there are enough system resources and all latency goals are met, the other workloads can burst up to their maximum limit.

Protect from rogue processes Even production quality software can spawn rogue processes that can consume an excessive amount of system resources. Intended actions like a database query executing full table scans during business hours can interfere heavily with transaction-based workloads and other interactive applications, resulting in high response times. Using Priority Optimization can prevent a process or an application from over-consuming resources during the day while lifting the limit during the night.

Manage application performance expectation levels When HPE sizes a storage system, it does so with the full expected workload for that array in mind. Workload performance is optimal when only a fraction of the intended workloads is active, such as during the initial deployment of the array. When the other workloads eventually come online, this performance level is not sustained. To avoid support calls from users about reduced performance when the remaining anticipated workloads are added to the system, an HPE 3PAR StoreServ system administrator should define a Priority Optimization QoS level for the initial workloads to make available a reduced set of array resources. This will set the right performance expectation level for users from the start.

Manage periodic variations in workloads Some I/O workloads occur periodically, such as daily backups, database consolidations, monthly payroll, and billing cycles. Such workloads may require significant I/O resources starting at a well-defined time for a known duration. Accommodating these workloads during business hours depletes the resources of interactive workloads and reduces user productivity. HPE recommends configuring periodic workloads with Priority Optimization limits for IOPS and bandwidth during times when the impact on production and interactive workloads would be unacceptable. As an example, backups could be configured to get their full throughput only during off hours. This prevents a backup that is incidentally executed during the workday from affecting daily production workloads.

Improving the multitenant and cloud experience HPE 3PAR StoreServ systems are well suited for multitenant environments and cloud providers who accommodate workloads from customers internal or external to the organization. In such an environment, each customer pays for the use of an agreed amount of disk capacity and array performance. HPE 3PAR StoreServ systems administrators can group objects in a virtual domain to isolate each customer’s workloads. With Priority Optimization, you can implement a QoS rule on an entire domain. This rule governs all current and future objects inside the domain and provides an efficient way to implement the performance level the customer bought.

Federated QoS with HPE 3PAR Peer Motion software HPE 3PAR Peer Motion software serves to migrate volumes online from one HPE 3PAR StoreServ system to another. The migration data traffic runs over a dedicated pair of Fibre Channel connections called “Peer links.” With up to nine virtual volumes (VVs) migrating concurrently, an HPE 3PAR Peer Motion operation can contend with other workloads for internal array resources on the source and destination system. Furthermore, the Fibre Channel ports on the source system that are accessed by the Peer links on the target system may be shared with other hosts. The large sequential reads on the source system from an HPE 3PAR Peer Motion operation may impact the traffic to and from other hosts using these same array host ports. Priority Optimization can be used on the source array to control the throughput of HPE 3PAR Peer Motion data transfer over the Peer links in a fine-grained way by enabling a QoS rule limiting IOPS or bandwidth to the VVsets holding the volumes under migration. This way, internal array resources and host port bandwidth are made available on the source array for competing workloads.

Virtual volume sets and virtual domains QoS rules operate on VVsets, virtual domains, and volumes that belong to neither of them. A VVset is an autonomic group object that is a collection of virtual volumes. VVsets help to simplify the administration of virtual volumes and reduce human error. An operation like exporting a VVset to a host will export all member volumes of the VVset. Adding a volume to an existing VVset will export that volume automatically to the host or host set the VVset is exported to. VVsets have several use cases beyond reducing the administration of their volume members, such as enabling simultaneous point-in-time snapshots of a group of volumes with a single command. Up to 8,192 volumes can be part of a VVset and up to 8,192 VVsets can be created. There is no requirement that volumes in a VVset are exported to a host for a QoS rule to be applied to the VVset.


Virtual domains provide a policy-based, secure administrative segregation of users and hosts within an HPE 3PAR StoreServ system but still share all system resources (ports, processors, cache, and disk drives) across all workloads. The result is a simple, efficient, and scalable approach with fine-grained access control that delivers secure logical isolation within a consolidated platform. You can create up to 1,024 virtual domains on an HPE 3PAR StoreServ system. A virtual domain set is an autonomic group object that combines two or more virtual domains that share their resources. This can be interesting when two virtual domains need to share a resource such as a backup service.

VVsets and virtual domains are created in the SSMC or in the CLI. Refer to the HPE 3PAR StoreServ Management Console 3.4 Administrator Guide, the HPE 3PAR StoreServ Management Console 3.4 User Guide, and the HPE 3PAR Command Line Interface Administrator User Guide for details.

Concepts and features Priority Optimization software introduces the features to manage and distribute the I/O capacity of an HPE 3PAR StoreServ system across multiple workloads in accordance with predefined specifications. The software enables the co-location of the workload data of different types such as sequential, random, and transactional, among others, with widely varying I/O request sizes on a single storage system while achieving adequate and predictable performance for every workload in a single or multi-tenant environment. Each QoS rule is associated with a single QoS target object. The smallest target object to which a QoS rule can be applied is a VVset. Because a VVset can contain a single volume, a QoS rule can target one virtual volume. A QoS rule operates on all volumes in a VVset, a virtual domain or all volumes that are not a part of either. A VVset cannot be removed unless all QoS rules acting on it are removed first. QoS rules are persistent across a reboot of an HPE 3PAR StoreServ system.

Starting from HPE 3PAR OS 3.1.2 MU2, Priority Optimization can cap performance of VVsets with an upper IOPS, bandwidth limit, or both. This cap is called the Max Limit and is the maximum amount of IOPS, bandwidth, or both that a given QoS object is allowed to achieve. HPE 3PAR System Reporter can be used to quantify the volume performance upon which acceptable QoS rules can be defined. HPE 3PAR OS 3.1.3 introduced the concepts of a Min Goal, a priority level (high, normal, and low), and a latency goal per QoS object along with the ability to define QoS rules against virtual domains. The Min Goal is the minimum amount of IOPS or bandwidth that the QoS subsystem will throttle a given QoS object in order to meet the latency goal of a higher-priority workload. If system resources are available, VVsets or virtual domains might consume more IOPS or bandwidth than the Min Goal but will be throttled if they reach their Max Limit. The performance might also be less than the Min Goal; this can happen if the application is requesting fewer IOPS or if the sum of all Min Goals defined is more than the I/O capability of the system or a given tier of storage.

Priority Optimization sets the Min Goal and Max Limit values for IOPS and bandwidth in QoS rules in absolute numbers, not in percentages. The IOPS number is stated as an integer between zero and 2,147,483,647 (231 - 1), although a more realistic upper limit is the number of IOPS the HPE 3PAR StoreServ is rated for. The bandwidth value is an integer between zero and 9,007,199,254,740,991 (slightly less than 263) expressed in KB/s, although a more realistic upper limit is the throughput in KB/s the HPE 3PAR StoreServ can handle over its I/O ports.

The latency goal of a QoS rule is the maximum value for the service time the system will try to provide for the applications governed by that rule. For the latency goal to work, rules with a Min Goal must exist so the system knows the minimum to which it can throttle other workloads to achieve the latency goal. Hierarchy in workloads is created by assigning them a priority level of high, normal, and low. As the system gets busier, it starts throttling lower-priority workloads first to meet latency goals of higher-priority workloads. In HPE 3PAR OS 3.2.2 the minimum latency goal was reduced from 1 to 0.5 milliseconds to accommodate the reduced service time seen on HPE 3PAR StoreServ all flash arrays. The latency goal ranges from 0.50 to 10,000 milliseconds.

Table 1 summarizes the controls of the QoS subsystem in an HPE 3PAR StoreServ system.

Table 1. Controls of the QoS subsystem in HPE 3PAR StoreServ

Control type Min HPE 3PAR OS Description Details

Max Limit 3.1.2 MU2 and later

Maximum threshold for IOPS or bandwidth for a QoS object

Max Limit has no dependencies on the other control types.

Min Goal 3.1.3 Minimum for IOPS or bandwidth below which Priority Optimization will not throttle a QoS object

When a Min Goal is set on an object the user must also configure a Max Limit on the same object within the same rule. Min Goal will be ignored if the system has no rules with latency goal set.

Latency goal 3.1.3 Service time target the system tries to achieve for the workload in a QoS object

This control type requires other rules in the system with Min Goal to be set because the latency goal algorithm needs direction regarding which workload to target and throttle. The order in which these are throttled is provided by the priority levels. Sub-millisecond latency goals are available since HPE 3PAR OS 3.2.2.

Priority level 3.1.3 Precedence order for the QoS subsystem to throttle workloads to meet latency goals

Use high priority against critical applications, lower priority on less critical applications.

https://support.hpe.com/hpsc/doc/public/display?docId=a00053598en_us



https://support.hpe.com/hpsc/doc/public/display?docId=c04204251



Mode of operation Priority Optimization operates on I/O communication between an HPE 3PAR StoreServ and a host. The QoS rules are defined for front-end IOPS and bandwidth and are applied via VVsets and virtual domains. When an I/O request reaches the HPE 3PAR StoreServ controllers, Priority Optimization will take one of the following three actions:

• Pass the I/O request to the VV layer of the HPE 3PAR OS

• Delay the I/O request by placing it in a private QoS queue on the HPE 3PAR StoreServ and process the queue periodically

• Return an SCSI Queue Full (QFULL) message to the server that sent the request

If the upper limit for IOPS or bandwidth for a VVset has been reached, Priority Optimization delays I/O request responses for the VVs contained in that VVset. These delayed I/O requests are pushed to an outstanding I/O queue for the volumes in the VVset experiencing the limit breach. Every QoS rule maintains its own queue for delayed I/O requests. These queues are constructed in the control cache of the HPE 3PAR StoreServ controller node receiving the I/O request that needs to be delayed. Only the I/O request (SCSI read or write request) descriptions are queued, not the actual data, leading to very fast handling of delay operations and no data cache consumption.

Delaying I/O requests is an attempt to force the server to throttle its workload to the storage array. When I/O requests reside longer than 200 ms in a QoS queue, any more incoming I/O requests to the volumes in the VVset are rejected and a QFULL response is returned to the host using the volumes. The QFULL response prevents delayed I/O from accumulating and eventually filling up system resources such as buffers and queues on the host, host bus adapter (HBA), and VV layer. The host application should respond appropriately to the QFULL message and throttle I/O. The I/O delay and the eventual QFULL response applies to all members of the VVset, even if only one of the volumes caused the QoS threshold breach.

During operation, Priority Optimization samples IOPS, bandwidth, and latency values per VVset and virtual domain every eight milliseconds. Priority Optimization assembles 625 of these time periods into a moving five-second window to adjust the ingress of the I/O requests to a given QoS rule on a periodic basis. The migration toward a new IOPS or bandwidth setting is driven by an exponentially weighted average of the five-second period and is completed in a few seconds. The impact of tens of QoS rules has been determined to be negligible for the control cache and the usage of the CPUs in the controller nodes.

QoS configuration with latency goals To deal with latency goals, HPE 3PAR OS 3.1.3 introduced the concept of the System Busy level in an HPE 3PAR StoreServ array. The value of the System Busy level is an internally computed scaling factor for controlling resources to Priority Optimization objects. With no latency goals present, the System Busy level remains at zero percent and all objects subject to a QoS rule experience only delayed or rejected I/O requests when reaching their Max Limit. When latency goals and Min Goals are defined in the system, the System Busy level is calculated in real-time from information whether or not QoS rules are meeting the latency goals.

If Priority Optimization detects that some QoS rules are not meeting their latency goals, it raises the System Busy level. If it detects that all QoS rules are meeting their latency goals, it lowers the System Busy level. The latency goal defined is assumed to be for an 8 KB I/O. As the System Busy level increases, this 8 KB I/O size is automatically scaled up linearly if the average I/O size is larger than 8 KB. The scaling factor is based on the latency goal set by the user and the average I/O size for the applications under the QoS rule. The scaling factor increases from one to (average_IO size / 8 KB) as the System Busy level goes up. For example, the latency goal value in a QoS rule is doubled for 16 KB average I/O size, tripled for 24 KB average I/O size, and so on. Latency goals of lower-priority QoS objects are scaled up earlier than those of higher-priority QoS objects. Due to the scaled latency goals, Priority Optimization starts throttling down lower-priority VVsets and virtual domains toward their Min Goal level, but higher priority rules continue to consume system resources to meet their specified latency goals. If the system continues to get busier, eventually even high-priority QoS objects are delayed I/O when the Min Goal values of all low and normal priority QoS rules are reached. Workloads are not throttled if Max Limit and Min Goal are equal.

Table 2 shows how the scaling multiplier is changed within the dynamic range of each priority level and the range of latency multipliers at the System Busy levels.


Table 2. System Busy level throttling

Rule priority System Busy level Latency goal scaling multiplier

Low 0% to 25%

25% to 50%

50% to 100%

From 1 to (average_IO size / 8 KB)

Average_IO size / 8 KB


Normal 0% to 25%

25% to 50%

50% to 100%

1



High 0% to 25%

25% to 50%

50% to 100%

1

1


Figure 2 shows an example of how different priorities are throttled as the System Busy level increases. A workload governed by a low priority rule is throttled first when latency goals are not met, followed by the normal priority workload and the high priority workload.

Figure 2. System Busy level throttling of objects depending on the priority of their QoS rule

Adaptation by the system to a modified latency goal for a QoS object takes place in a few seconds.

0 25% 50% 70% 100% Busy level

Min goal 5kMin goal 4kMin goal 3k

IOPS

Max limit 12k

Max limit 8k

Max limit 6k

High priority ruleNormal priority ruleLow priority rule

High

Normal

Low


QoS rule attributes Every QoS rule has these eight attributes:

• System: The name of the HPE 3PAR StoreServ system the QoS rule is working on

• Name: The name of the QoS rule (fixed to the name of the VVset or the virtual domain)

• Scope: The target object ruled—can be a VVset, a virtual domain, or all volumes not part of either (System rule)

• State: The QoS rule can be active or disabled

• IOPS range: The Min Goal and Max Limit on IOPS for the target object

• Bandwidth range: The Min Goal and Max Limit on transfer rates for the target object

• Latency goal: The maximum service time for the target object

• Priority: The priority level for the target object

QoS rules can be created by the HPE 3PAR CLI, SSMC, and the WSAPI. The rule can be active or inactive at creation time.

The “all_others” System rule Priority Optimization features a built-in rule of target type System that is called “all_others.” The System rules allows you to specify a Min Goal and Max Limit for IOPS and bandwidth, a priority level, and a latency goal for all volumes and VVsets in the system that are not subject to a named QoS rule. The System rule is disabled by default. Enabling this rule eliminates the need to define a named rule for every volume on the storage system.

QoS rule actions QoS rules are subject to five distinct actions:

• Create: A QoS rule is created.

• Enable: A disabled QoS rule is made active.

• Disable: An enabled QoS rule is made inactive.

• Change: The limit values, latency, or priority for an existing QoS rule are modified.

• Clear: The QoS rule is removed.

The action diagram for QoS rules is shown in Figure 3. The possible actions are shown with an orange outline and the result of an action is shown in a dashed blue outline.


Figure 3. Action diagram for QoS rules

Changing a limit or latency value in a QoS rule activates the value instantly on the object; there is no need to disable the QoS rule first. An active QoS rule can be removed without first disabling it.

Overlapping QoS rules A VV can be a member of up to eight VVsets that each can have a QoS rule defined. In such a case, the I/O to and from the volume is governed by multiple, possibly overlapping rules. In this case, all QoS rules on the VVsets that the volume is a member of are combined into a new rule in which the lowest Max Limit, the lowest Min Limit, and the lowest latency goal apply to the volume.

In case the volume is a member of a VVset with QoS Rule A and a VVset without a rule or with a disabled rule, the System rule limits are superimposed onto Rule A. In case the volume is a member of a VVset with a QoS Rule A and a VVset with QoS Rule B that has no numerical limit defined, Rule A applies; the System rule is ignored.

QoS on virtual and physical copies Virtual and physical copies of volumes in a VVset are not automatically members of the parent VVset. You can add them manually to the parent VVset in order to be governed by the same QoS rule or you can add them to another VVset with a different QoS rule.


Managing Priority Optimization For creating, managing, and removing QoS rules for Priority Optimization, HPE offers the SSMC, the HPE 3PAR CLI, and the WSAPI. The use of each of these interfaces is discussed in this section.

Using the HPE 3PAR StoreServ Management Console To create a new QoS rule for a VVset using the SSMC, click the large menu button on the top left of any SSMC screen, select the Show all link on the right of the menu opening, and click the Priority Optimization link, as shown in Figure 4.

Figure 4. The Priority Optimization link in the SSMC menu

The list of QoS policies present in the systems within scope now appears.

Figure 5. The list of Priority Optimization rules with their details organized by target type

Clicking the green Create priority optimization policy button opens the popup screen in Figure 6 to create a QoS rule and options. Rules created can be enabled or disabled at creation time. If the rule exists already, highlight it as shown in Figure 5 and click Actions → Edit to make changes to it.


Figure 6. Screen to create a new Priority Optimization policy

The target type in Figure 6 for a new rule can be a VVset, a domain, or a system, as shown in Figure 7.

Figure 7. Selecting the target type for a new Priority Optimization policy


Selecting the magnifying glass at the Target name field as shown in Figure 8 lists the VVsets or domains present in the system that are not yet a member of a QoS policy.

Figure 8. Screen to select an existing VVset for a QoS rule

The target type of domain forces the QoS rule onto all current and future members of the domain specified for that target name. When you specify a rule for a VVset or domain, the object must already exist.

The system target type shown in Figure 7 has the predefined target name all_others. The System rule governs all volumes that are not subject to an explicit QoS rule or that has a rule that is disabled. Be careful when defining the limits for the system target type because the number of volumes managed by this rule can be large. To ensure that at least some IOPS and bandwidth is available to the volumes under this rule, Priority Optimization enforces a minimum value for minimum goal of 1,000 I/O requests per second and 100,000 KB/s for the Min Goal for bandwidth. These limits might be far too low for some environments. If lower values are entered, a popup displays information about the minimum values.

The optimization parameters for a QoS rule are shown in Figure 9.

Figure 9. Optimization settings for a QoS rule

Every rule has minimum of one and maximum of six optimization parameters. Priority is a mandatory parameter with values low, normal, or high, as shown in Figure 10.


Figure 10. Selecting the Priority level for a QoS policy

To make a rule operational, you should at minimum specify an IOPS Max Limit, a bandwidth Max Limit, or a latency goal. A rule with only a priority is ignored unless the system has one or more rules with a latency goal and a Min Goal present. For IOPS and bandwidth, a Min Goal cannot be specified without a Max Limit. When only a Max Limit is specified, the Min Goal is made equal to it. The Max Limit must be equal to or higher than the Min Goal value.

Figure 11 shows the rule with target name “Exchange2019” with its optimization options in the right pane. The Edit option in the Actions menu on the right-hand side of the figure enables you to make modifications to any option, to disable or enable the rule, or to delete it.

Figure 11. QoS policy example in the SSMC

When a QoS rule is disabled, the IOPS and bandwidth consumed by the volumes in the VVset may go up if they were limited by the rule unless the System rule is enabled, enforcing its limits on the VVset. Changes to a QoS rule are implemented immediately to the target object.

Figure 12 shows the list of QoS rules with their details exposed for various systems. Targets Meghan and Exchange2019 are embedded in a VVset on system peche with a QoS rule with latency goals among others. Target Patent_tracking is embedded in a domain on system aperol and has a QoS rule based on I/O bandwidth. The all_others target on system melba governs all volumes and VVsets that have no QoS rule defined or whose rule is disabled and is governed by an IOPS Min Goal and Max Limit.


Figure 12. QoS policy examples with details expanded

The columns in Figure 12 can be sorted in alphanumerical order, one column at a time.

Using the HPE 3PAR Command Line Interface You can also create and manage QoS rules in Priority Optimization from the HPE 3PAR CLI. The three CLI commands to create and manage QoS rules are setqos, showqos, and statqos. Refer to the help page for each of the commands in the HPE 3PAR CLI and to the HPE 3PAR Command Line Interface reference for details about the command syntax and explanations for the different columns in the output. The integer value for bandwidth in the setqos command can optionally be followed with “k” or “K” to indicate a multiple of 1,000, with “m” or “M” to indicate a multiple of 1,000,000, or with “g” or “G” to indicate a multiple of 1,000,000,000. If none is specified as the value for IOPS, bandwidth, or latency, the limits for it are removed. Here is an example of the creation of a QoS rule for a VVset called “ESXi_60”:

setqos vvset:ESXi_60 -io 6000-18000 -bw 100M-300M

The output of showqos after this command was executed is displayed in Figure 13.

Figure 13. Sample output of the HPE 3PAR CLI command showqos

Rules created by the HPE 3PAR CLI can be managed by the SSMC and vice versa. You can keep a QoS rule created with setqos inactive at creation time by using the –off option. All named rules for VVsets can be switched off simultaneously with the command setqos –off vvset:*, and on again with setqos –on vvset:*. The same is true for domains. The built-in rule all_others is switched on and off with the command setqos [-on|–off] sys:all_others.

You can show rules in ordered form with key column X by using the command showqos –sortcol X, [inc|dec] where X is the column number starting with the left column numbered 0. The options inc and dec sort on increasing or decreasing values for column X. Columns with alphanumerical content are sorted alphabetically. Figure 14 shows the output of the showqos command with the sorting option for an increasing IOPS Max Limit value.



Figure 14. List of all QoS rules in the system ordered on increasing IOPS Max Limit value

Using the pattern option with the setqos command changes the QoS rule for all VVsets that include VVs with names that match any of the names or patterns specified. The syntax is –vv <name>|<pattern>. If the volume is in a VVset that had no QoS rule yet defined, a new rule is created for the VVset with the limits specified.

Any role that is granted the qos_set permission can define QoS configurations. An account with a Super or Edit role has this right assigned and can execute QoS CLI commands. You can schedule QoS rules by using the createsched command for automated QoS policy changes based on time of the day, day of the week, and other parameters. In this way, QoS rules can adapt to day or night or weekday or weekend workload conditions.

The srstatqos command was introduced with HPE 3PAR CLI 3.1.2 MU2. This command extracts and displays historical performance reports on the QoS rules from the HPE 3PAR System Reporter database on the system’s controller nodes. HPE 3PAR System Reporter 3.1 MU1 and later installed on a separate server also captures all statistics on QoS.

Using the Web Services API The WSAPI for HPE 3PAR StoreServ is integrated into HPE 3PAR OS 3.1.2 and later and offers an API defining the operations, inputs, and outputs supported. The WSAPI interface provides a flexible, fast, and powerful way to perform storage management tasks. It enables automated management of hosts, ports, VVs, common provisioning groups (CPGs), virtual logical unit numbers (VLUNs), storage space, QoS rules, and system information. WSAPI clients interact with the WSAPI server on the HPE 3PAR StoreServ system through HTTP or HTTPS. The free software development kit (SDK) 1.6.3 for WSAPI for HPE 3PAR StoreServ includes bindings for Java and Perl to develop command line scripts and GUIs to create, modify, remove, and query every feature of QoS objects. For more information, refer to the HPE 3PAR Web Services API Developer’s Guide.

HPE 3PAR StoreServ software interoperability Priority Optimization sets and manages QoS rules defined for I/O traffic from servers connected to the front-end host ports of an HPE 3PAR StoreServ array. Software titles for HPE 3PAR StoreServ systems, such as Dynamic Optimization, Adaptive Optimization, Virtual Copy, System Reporter, Thin Provisioning Suite, Data at Rest Encryption, and Recovery Manager, move data from and to the back-end of the array. This means that they are all compatible with and operate transparently with Priority Optimization. Peer Motion can be the subject of a QoS rule on the source HPE 3PAR StoreServ system. Peer Motion and Online Import traffic to the destination HPE 3PAR StoreServ system are not throttled by any QoS rule including the all_others rule.

Application interoperability External applications exhibit strong use cases for their integration with Priority Optimization. A few of these use cases are illustrated in this section.

Databases Many customers consider databases to be a mission-critical application. Careful performance planning allows for the creation of a QoS policy that helps ensure the proper day-to-day operation of a database and addresses runaway queries that might consume a large amount of IOPS and bandwidth.

All database software vendors recommend that data files, index files, and transactional and archive logs be kept on separate volumes. This allows for individual QoS rules on each of the database component files. The write capability of online transaction logs is most important because the entire performance of the database depends on it. For this reason, a QoS rule on the volumes containing the online transaction logs should be carefully dimensioned so that the performance of the database is not inhibited.

https://h20392.www2.hpe.com/ecommerce/efulfillment/getReceipt.do?lc=EN_US




Mail systems Email represents a mission-critical workload in nearly every organization. Browsing and composing mail is a highly interactive operation, and users demand a swift response to mouse clicks when using their mail client software. From an architectural standpoint, mail software has been traditionally implemented on dedicated servers connected to dedicated storage systems. The availability of more powerful hardware started a consolidation trend in which email software shares the server and the storage system with other applications. HPE 3PAR StoreServ systems are well suited as mail servers because they deliver excellent interactive response in the presence of other applications. As a general practice, it is recommended that the mailbox database files and the log files be kept on separate volumes. The databases can be spread over multiple volumes combined in a VVset. Using a QoS rule to limit the IOPS or bandwidth to the VVsets for the mail database of a group of users enhances the performance of email for other users and for other applications on HPE 3PAR StoreServ systems. Implementing a latency goal is a suitable way to enforce a good response time for the email system.

Virtualization software Virtualization platforms such as VMware®, Microsoft® Hyper-V®, and Citrix® XenServer® execute business-critical and latency-sensitive applications in parallel with other less-critical applications. It is important to prioritize I/O requests from important applications over the requests from less important applications. The virtualization software creates and manages virtual data stores that are container files comprising one or more virtual drives used by virtual machines (VMs). Each container file is built on one or more storage LUNs that reside on the storage array and are usually accessed over the SAN. Applying a Priority Optimization QoS rule to the LUNs that make up a logical storage container will control the IOPS or bandwidth for all VMs whose virtual drives are contained in that LUN. Be sure that there is enough bandwidth and IOPS in the QoS rule so that the applications running on the VMs can deliver acceptable I/O response times.

A QoS rule on a VVset making up a VMware vSphere® datastore puts a limit on the I/O requests for all VMs using that datastore. This level of control is not granular enough if you want to grant some VMs more I/O resources than others in case of contention. Recent versions of vSphere offer three native types of granular I/O resource control. Table 3 lists the vSphere characteristics together with those of Priority Optimization QoS.

Table 3. Characteristics of Priority Optimization and VMware I/O resource control mechanisms

HPE 3PAR QoS vSphere Storage I/O Control (SIOC)

vSphere Adaptive Queue Depth (AQD)

vSphere Storage Distributed Resource Scheduler (DRS)

I/O control technique Set Min Goal (3.1.3) and Max Limit for IOPS and bandwidth, set latency goal (3.1.3), set priority level (3.1.3)

Enforce predefined I/O shares for each VM

Control queue depth of data store SAN LUN in VMkernel

Migrate VM to other data store

Reacting on System Busy level growing, Max Limit reached

I/O latency growing Queue Full or Device Busy at LUN or port level

I/O latency and space utilization growing

Granularity VVset, virtual domain, system All VMs in a datastore All hosts using the SAN LUN for the datastore or a particular port on the HPE 3PAR StoreServ system

Single VM

Managed from HPE 3PAR SSMC/HPE 3PAR CLI/HPE 3PAR WSAPI

vSphere/CLI vSphere/CLI vSphere/CLI

Available in HPE 3PAR OS 3.1.2 MU2 and later

vSphere 4.1 and later with Enterprise Plus license

vSphere 3.5U4 and later with Standard license

vSphere 5.0 and later with Enterprise Plus license

Combining QoS with SIOC offers I/O control to the level of an individual VM—an I/O share and an optional IOPS limit defined per VM distributes the available I/O capacity allotted by QoS to the datastore in a customized way to the VMs. Note that SIOC does not respond to Queue Full messages from QoS directly. SIOC is not supported for RDM disks so Priority Optimization fills in when IOPS and bandwidth management is required.

Priority Optimization can also cooperate with the AQD algorithm to manage I/O. AQD adjusts the LUN queue depth in the VMkernel I/O stack when the storage array indicates I/O congestion by returning a BUSY or QUEUE FULL status. These status codes may indicate congestion at the LUN level or at the ports on the array. When congestion is detected, VMkernel throttles the LUN queue depth. This reduction in the queue depth gives the array the opportunity to decrease the number of outstanding I/O requests. The VMkernel attempts to gradually restore the queue depth when congestion conditions decrease. With vSphere 5.1 U1 and later, you can configure AQD per LUN or globally for the VMware ESXi™ host.

You can combine Priority Optimization with SIOC and AQD to provide I/O congestion control on three different levels. A workload’s I/O requests will first observe its QoS limits set on the HPE 3PAR StoreServ array. The I/O requests will next be impacted when SIOC reacts to latency increases and when AQD responds to Queue Full messages from QoS. Coordination and some experimenting may be necessary to achieve the best results.


VMware Storage Distributed Resource Scheduler (DRS) is a feature introduced in vSphere 5.0. Storage DRS allows for automated or manual load balancing of virtual machine disks (VMDKs) across data stores on multiple ESXi hosts based on I/O latency and space utilization. When Storage DRS migrates a VM’s VMDK, I/O decreases on the original datastore but increases on the target one. As a result, combining Priority Optimization with Storage DRS requires careful QoS rule planning. The QoS rule on any potential Storage DRS destination data store should have enough headroom to accommodate the additional I/O capacity of the migrated VMDK. This can lead to a managerial challenge in larger ESXi clusters that can be addressed by defining a set of affinity rules for each VM to reduce the number of target datastores for it during a DRS migration. If the workload’s I/O characteristics of a VM are not well known, HPE recommends using manual Storage DRS migrations instead of automated ones to help detect any I/O congestion that occurs after the migration process. When you use DRS to move datastores, HPE does not recommend also setting a QoS latency goal on the VVset that holds the datastore.

With Microsoft Windows Server® 2012 R2 and later, Hyper-V includes a feature named Storage QoS. This feature allows storage IOPS performance isolation in a multitenant environment for a virtual hard disks (VHDs). Storage QoS shares the concept of SIOC. It can restrict disk throughput for less important virtual machines to allow more important ones to attain their service level. Two settings are offered, both of which default to zero (meaning unmanaged):

• Minimum: Specifies the minimum number of IOPS that a VHD can draw from the physical storage

• Maximum: Specifies the maximum number of IOPS that a VHD can draw from the physical storage

The minimum IOPS value provides a specified level of service to the workloads on the VHD when I/O congestion occurs and permits more IOPS by workloads in circumstances where there is no congestion. The minimum and maximum settings can be applied and modified while the VM is running and are activated immediately. Maximum and minimum values are specified in terms of normalized IOPS where every 8 KB of data is counted as an I/O. Note that Storage QoS is not supported for Hyper-V pass-through disks or shared disks. Storage QoS is a feature of the Hyper-V host, so the guest operating system is irrelevant.

Priority Optimization can be combined with Hyper-V QoS management. Priority Optimization manages the overall I/O, latency, and priority to the LUN on the HPE 3PAR StoreServ array. Hyper-V QoS manages the IOPS made available between the VMs installed on the LUN.

OpenStack Over the past several years, HPE has contributed significantly to the advancement of the OpenStack® project. HPE developed HPE 3PAR StoreServ block storage drivers, which support the OpenStack technology across both iSCSI and Fibre Channel protocols. These drivers are based on the Block Storage Cinder plug-in architecture. The drivers execute volume operations by communicating with the HPE 3PAR storage system over HTTP or HTTPS and Secure Shell (SSH) connections. The HTTP, HTTPS, and REST communications use the hp3parclient Python package to talk to an HPE 3PAR StoreServ system.

QoS to the HPE 3PAR VVsets with Max Limits for IOPS and bandwidth was introduced with OpenStack Havana. Starting with OpenStack Ice House, a user can set Min Goal or Max Limit values for IOPS and bandwidth and latency goals per VVset, per domain, and for the System rule. Refer to the white paper titled OpenStack HPE 3PAR StoreServ Block Storage Drivers Configuration Best Practices for more information.

Reporting Continuous monitoring of all critical parameters is essential when managing a storage system. SSMC features reports for monitoring resources related to Priority Optimization graphically over time. The charts in the reports cover data on IOPS, bandwidth, service time, wait time, I/O size, rejected I/O requests, and queue length in high, medium, or low resolution for a selectable number of hours, days, or months for the entire array, for one or multiple VVsets or for domains. The charts can be exported to a CSV file or in PDF format. No System Reporter license is needed for these reports. Figure 15 shows a series of high-resolution graphs of the I/O behavior of an HPE 3PAR StoreServ system over a period of 24 hours in the SSMC. The workload to the VVsets was synthetic.

The HPE 3PAR Excel add-in for SSMC 3.4 enables you to extract and manipulate data from the HPE 3PAR StoreServ System Report database in Microsoft Excel. The add-in extracts data using the SSMC server, and provides a convenient way to create custom reports on performance for a wide range of storage array objects.

The HPE 3PAR CLI command statqos displays live statistics of active QoS rules. The command produces an averaged output every two seconds by default (this interval can be changed with the –d option). The command displays per active QoS rule the values for IOPS, bandwidth, service time (Svt_ms), wait time (Wtt_ms), size of the I/O requests (IOSz_KB), number of rejected I/O requests (Rej), average QoS queue length (Qlen), and average wait queue length (WQlen). Figure 16 shows a sample output for this command.

https://docs.microsoft.com/en-us/windows-server/storage/storage-qos/storage-qos-overview

https://www.hpe.com/us/en/product-catalog/storage/storage-software/pip.hpe-openstack-for-3par-storeserv-and-storevirtual-software.1008537377.html

https://pythonhosted.org/hp3parclient/index.html

https://h20195.www2.hpe.com/v2/getdocument.aspx?docname=4aa6-5209enw

https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00060613en_us&docLocale=en_US


Figure 15. Graphical representation in the SSMC of the history of I/O resources in use on an HPE 3PAR StoreServ Storage


Figure 16. Output of the statqos command for four iterations of five seconds each

You can find an explanation of the column headers of the statqos command in the HPE 3PAR Command Line Interface Administrator Guide.

The on-node System Reporter tool collects and stores QoS statistics periodically for all active QoS rules. The HPE 3PAR CLI command srstatqos reports statistics for the QoS rules for one or multiple VVsets, domains, or for the System rule over any timeframe. The options to srstatqos mirror other srstat* commands. The default output of srstatqos shows an averaged data point per five minutes. Figure 17 shows the average IOPS, bandwidth, and some other parameters for the entire array for the current day from noon until 1 PM.

Figure 17. High-resolution QoS output from srstatqos for a defined time period for the entire array

Information by VVset and by domain over the same time period is shown in Figure 18.

Figure 18. QoS information from srstatqos by VVset, by domain, and for System for a defined time period




Refer to the information on statqos in this white paper and on the help page for srstatqos for an explanation of the column headers shown in the table of Figures 17 and 18. The srstatvlun command allows you to filter the collected high-resolution information by host or VVset. This is shown for the last hour in Figure 19.

Figure 19. High-resolution QoS information from srstatvlun filtered by host and VVset for the last hour

Event and alert management HPE 3PAR OS 3.1.3 introduced a new set of CLI commands to create and manage performance alerts. SSMC has a template for alerts in the Threshold Alerts section of System Reporter. The alert conditions are executed against the on-node System Reporter database.

The choice of the condition criteria upon which an alert is generated is rich and versatile. If a criterion or a combination of them is met, the alert framework generates an alert message and forwards it using Simple Network Management Protocol (SNMP) traps or email if configured. An alert can be generated for the QoS targets being a VVset, a domain, and System. The following example used the CLI to generate a major type of alert in case of rejected I/O requests for the QoS object of domain DB_Production:

createsralertcrit qos -hires -major -target domain:DB_Production rej>0 Rejected-DB_Production

The hires option in this command evaluates this criterion periodically looking back over a five-minute interval. Alternatively, the criterion can be evaluated on an hourly basis or on a daily one (at midnight). The severity of the alert is chosen from major, minor, and info. The command showsralertcrit displays the list of alerts on the system, removesralertcrit removes existing alerts, and setsralertcrit disables or enables an alert and makes modifications to existing ones. A large number of options and specifiers exist for these commands. Consult the help page for the command in HPE 3PAR OS for more information or use the templates in the SSMC.

The following output of the HPE 3PAR CLI command showalert lists a major alert generated by the createsralertcrit command:

Id : 48 State : New Message Code : 0x0140004 Time : 2018-09-10 15:20:02 CET Severity : Major Type : System Reporter QoS performance alert Message : System Reporter alert criterion Rejected-all_others with condition rej>0 has

been satisfied by qos 16 with value(s) 24.0

The fragment qos 16 in the line starting with Message points to the sequential number for the QoS rule. This number can be found in the output of the showqos command. The number 24.0 in the Message line is the amount of rejected I/O requests that were registered for the rule within a period of five minutes. Note that the alerts list is cleared at the beginning of every new measuring period. Alert processing requires the presence of a System Reporter license on the system.


You can also monitor rejected I/O requests by looking at the Rej column in the output of the srstatqos command. Alternatively, you can review the debug event logs for QFULL messages that occurred the last 24 hours by using the command showeventlog –debug –msg Qfull –min 1440. Here is an example of a QFULL event from the debug event log:

Time : 2018-09-13 14:04:23 CEST Severity : Debug Type : Host error Message : Port 0:1:1 -- SCSI status 0x28 (Qfull) Host:machine.bel.hp.com (WWN 50060B0000C21639) LUN:2 LUN WWN:60002AC000000000000000AC000005D7 VV:0

CDB:28002692AAF000009000 (Read10) Skey:0x00 (No sense) asc/q:0x00/00 No additional sense information) VVstat:0x00 (TE_PASS -- Success) after 0.000s (-) toterr:9720, lunerr:2

Best practices Assembling VVs into VVsets and virtual domains VVsets are a mechanism for grouping HPE 3PAR VVs logically so that a common action can be applied to all volumes simultaneously. Applying VVsets reduces administrative overhead and the chance for human error. Examples include creating snapshots of a set of VVs autonomically or exporting several VVs to hosts as a group. Another use case for VVsets is the deployment of Priority Optimization. QoS rules are defined on VVsets, not on volumes, so the use of VVsets is required when setting up Priority Optimization. Volumes can be placed into VVsets without a service interruption. Note that using QoS to control I/O requests to a VVset containing VVs of disparate workloads, such as a VMware and other workload, might lead to unpredictable performance for the workloads. In this case it is better to create two or more VVsets, each with their own QoS rule.

HPE 3PAR virtual domain software is a solution for creating virtual private arrays (VPAs) within a single HPE 3PAR StoreServ array. With virtual domains, a user can be granted administrative rights for volumes and host within a domain in without being able to view or change array resources outside of the assigned domain. This strict segregation of domain resources provides security and ease of management. Starting with HPE 3PAR OS 3.1.3, QoS rules can be laid out on virtual domains managing I/O requests for all its domain member volumes. Creating virtual domains can simplify QoS management for multi-tenant environments where array users pay for the resources used. HPE recommends using virtual domains when possible. Domains can be assembled in domain sets for an even higher level of abstraction and ease of management. Volumes and hosts can be moved to virtual domains, and virtual domains can be assembled into virtual domain sets without service interruption.

Determining the IOPS and bandwidth capacity for a system QoS rules in Priority Optimization define limits for IOPS and bandwidth for VVsets, virtual domains, and system in absolute numbers. Because of this, QoS administrators need accurate data on a system’s maximum IOPS and bandwidth capability. Using sizer tools, HPE can estimate the maximum front-end IOPS and bandwidth capability for a system when assuming a particular I/O request size and a given ratio for read/write (R/W) requests. Re-analysis of the system’s capability followed by the derivation of new values for the Max Limit, Min Goal, and latency goal for existing QoS rules should be considered whenever major changes occur. Examples include when:

• Upgrading the HPE 3PAR StoreServ system with additional disk drives or controller nodes

• Configuring Adaptive Optimization on the system or when moving an application to a new disk hardware tier (for example, from Nearline drives to Fast Class or the reverse)

• Connecting additional physical hosts or virtual machines with extra workloads to the storage system

• Upgrading the host hardware to a newer generation (for example, upgrading from HPE ProLiant Gen8 to Gen10) increasing IOPS and bandwidth load on the array

Setting up QoS rules IOPS and bandwidth consumption for existing workloads can be determined accurately by using HPE 3PAR System Reporter. When crafting a service-level agreement (SLA) for uncharacterized applications or when deploying workloads for the first time on an HPE 3PAR StoreServ, consider factors beyond the purely quantitative metrics for IOPS and bandwidth. Performance estimation involves several variables such as R/W ratios, I/O request block size, sequential or random access, and percentage of cache hits. Accounting for all these variables requires a comprehensive approach. An understanding of these variables is required before setting up and applying QoS rules for applications.


When starting with uncharacterized workloads, you can apply the following scenario to introduce a QoS rule for each workload:

1. Embed the storage volumes for the application that is estimated to require the most resources on the HPE 3PAR StoreServ in a VVset and an optional domain.

2. Based on external information or System Reporter data, create and activate a QoS rule with Max Limits for IOPS and/or bandwidth for the application. Optionally add Min Goals and a latency goal and set the priority to high. Watch the performance of the application and tune its QoS attributes if deemed necessary.

3. Repeat Step 2 for all the VVsets and domains for other applications that need a QoS rule in order of decreasing resource needs. Watch all applications’ performance that received a QoS rule and adjust their QoS limits if needed.

4. When Step 3 is complete, create the System rule with reasonable limits to make sure that applications and domains without a named QoS rule will not impact those with one.

Best practices on the collection and interpretation of HPE 3PAR System Reporter data are beyond the scope of this paper. For more information on this, consult the HPE 3PAR External System Reporter Software User Guide, the HPE 3PAR StoreServ Management Console 3.4 User Guide, and the HPE 3PAR System Reporter for SSMC technical white paper.

Implementing the System rule Workloads on an HPE 3PAR StoreServ Storage array that are not managed by a QoS rule can consume I/O resources indefinitely and end up starving other important workloads that share the array. To ensure that a controlled level of performance is in place across tenants and applications, all volumes must be governed by a QoS rule with meaningful limits reflecting the application’s minimum and maximum IOPS and bandwidth requirements. To avoid the scenario that every object on an HPE 3PAR StoreServ array needs its own QoS rule, Priority Optimization includes the built-in System QoS rule that controls I/O traffic to and from VVs, VVsets, and domains that are not subject to a named QoS rule or subject to a QoS rule that is disabled.

HPE recommends enabling the System rule with meaningful values to prevent any new volumes added to the HPE 3PAR StoreServ system or existing ones that are not subject to a QoS rule from negatively affecting the entire system by overconsuming IOPS or bandwidth.

The minimum allowed values for Max Limit for the System rule are 1,000 for IOPS and 100,000 KB/s for bandwidth. When setting up QoS on an existing system for the first time, be sure that there are enough resources left for the objects governed by the System rule after all named rules were created. Note that when a named QoS rule is disabled, it will be subject to the limits of the System rule if enabled.

HPE recommends that a named QoS rule for a workload be created and enabled before starting the workload. This way the workload will not be governed by the System rule with Max Limits for IOPS and bandwidth that might be far too small for adequate workload performance.

Handling Tier 1 applications HPE 3PAR StoreServ systems are Tier 1 arrays that can handle multiple mission-critical workloads concurrently next to workloads that have medium importance and performance requirements. Tier 1 applications should be given all the resources they need during runtime given their vital business importance. Profiling applications to determine their IOPS and bandwidth consumption can be done with HPE 3PAR System Reporter.

HPE recommends that explicit QoS rules be defined with sensible limits for IOPS and bandwidth on all VVsets used in Tier 1 applications. The System rule should always be defined and enabled to control I/O to applications on the system without an explicit QoS rule to prevent them from impacting Tier 1 applications. From HPE 3PAR OS 3.1.3 on, a priority level and a latency goal can be configured for QoS objects. Three priority levels are available, in descending order of precedence: high, normal, and low. The latency goal, expressed in milliseconds, is the maximum service time desired for a QoS object. Mission-critical applications should be given a latency goal and a high or medium priority. If Priority Optimization detects an application is not meeting its latency goal, the array’s System Busy level will increase resulting in the upscaling of latency goals of low priority workloads and, if needed, of normal priority workloads. Workloads without latency goals will be throttled as well, starting from the lowest priority ones. The resources freed up by this upscaling will allow Tier 1 applications to regain their performance level and meet their latency goal. Users may choose to set the Max Limit equal to the Min Goal if they do not want these applications to be throttled when the System Busy level increases.

Setting correct latency goals The System Busy level for the array is calculated from the current measured latencies and the latency goal value for the QoS rules. Be careful when determining the latency goal for a QoS object. If the latency goal is set too low for an application, the System Busy level will go up and lower priority VVsets and domains will be unnecessarily throttled, even when the load is light. If latency goal is set too high, the System Busy level will not go up and no volumes will be throttled, resulting in the service times increasing, impairing the application’s performance. The tier on which the volumes object of a QoS rule resides on should be considered when setting a latency goal. Some guidelines are listed in Table 4.

http://h20628.www2.hp.com/km-ext/kmcsdirect/emr_na-c03606466-1.pdf



https://h20195.www2.hpe.com/V2/GetPDF.aspx/a00038501enw.pdf


Table 4. Latency goal guidelines for disk tiers

Tier Latency goal guidelines

Nearline >= 20 ms

Fast Class >= 10 ms

SSD >= 0.5 ms

Sub-tiered volumes (Adaptive Optimization): Guideline for the middle tier in a three-tier AO configuration or of the lowest tier in a two-tier AO configuration

When a latency goal is set on a QoS object in a given tier, users should also create QoS rules with a Min Goal and a lower priority on other objects that reside in the same tier. This will allow QoS to throttle VVsets that share the same tier toward their Min Goal if there is contention for resources on that tier of storage in order to meet the latency goal of the higher-priority workloads.

HPE 3PAR Adaptive Optimization and QoS can cooperate, but HPE 3PAR Adaptive Optimization data migration may impact latency goal settings. For example, when a workload is partially migrated from Fast Class drives into Nearline drives by HPE 3PAR Adaptive Optimization, the latency goal that was set for performance on Fast Class drives might be impossible to meet on Nearline drives. This forces other workloads to be throttled towards their Min Goal, an activity with potentially a large impact. HPE recommends adapting the latency goal when submitting workloads with a QoS rule to Adaptive Optimization.

Setting Min Goals Priority Optimization in HPE 3PAR OS 3.1.3 and later enables you to set a Min Goal for IOPS and bandwidth for VVsets, virtual domains, and system. This is the floor below which the system will not throttle the QoS object when the Priority Optimization subsystem tries to meet the latency goal of a higher-priority workload. The Min Goal is not a guaranteed minimum performance level for an application: if the sum of all Min Goals defined is higher than the I/O capability of the system, some Min Goals will not be met.

The Min Goal and the latency goal for a VVset can influence each other. Assume a host application is sending I/O to volumes in a VVset on Nearline disks. The QoS rule for the VVset has a latency goal set to 20 ms and the Min Goal is set to 1000 IOPS. The actual latency measured by QoS is 12 ms. Assume the throughput of the application is 700 IOPS, which is less than the Min Goal of 1000 IOPS. Because the measured latency is lower than the goal of 20 ms, QoS will not throttle I/O in other VVsets to improve the performance of this one. This makes it clear that Priority Optimization is tuned to meeting latency goal values as a first concern.

Using alert criteria to monitor QoS-related metrics QoS rules impose bounds to the I/O completion of applications. When an upper limit is met, Priority Optimization delays processing the I/O request coming from the application’s host. This results in an increase of latency for applications on the volumes exported to that host. Delaying I/O requests is a lightweight activity on the HPE 3PAR StoreServ array. Rejected I/O requests incur processing activity on the host and impact the application because it will be throttled. Storage administrators should set up a process for monitoring their systems daily for rejected I/O requests to ensure the proper balance between I/O needs and QoS limits defined. This practice aids the performance of the applications and the storage system. The CLI command createsralertcrit introduced in HPE 3PAR OS 3.1.3 combined with the output of showeventlog and srstatqos enables you to set up a mechanism for detecting rejected I/O requests from multiple angles. SSMC sets up these alert criteria and metrics in a graphical way.

Using the latency goal to accommodate bursts in workloads Defining a QoS rule with a Max Limit for the IOPS or bandwidth an application prohibits a workload from bursting over the limit. For this purpose, Priority Optimization introduced a latency goal in a QoS rule for a VVset, virtual domain, and system. You can think of the latency goal as the hard minimum of performance for an application. Without a Max Limit present in the rule, the application can consume all resources needed to meet the latency goal. If the goal is not met, Priority Optimization pulls I/O resources away from other, lower-priority workloads and allocates them to the underperforming one until the measured latency matches the goal.


Figure 20. Graphical representation of the effect of a latency goal on a Tier 1 application to the IOPS consumption and service time of other workloads in an HPE 3PAR StoreServ system

Figure 20 illustrates the effect of a workload with a latency goal and other workloads without one. The horizontal axis in each of the two graphs is time in minutes, the vertical axes are IOPS consumed and service time per workload. Because the timeline in each graph is identical, data points in graphs can be compared vertically across the graphs. The purple line in the graphs is for Workload 1 that has QoS rule defined for a latency goal of 6 ms, a high priority and no Min Goal or Max Limit. The steady state IOPS consumption for Workload 1 is approximately 4000 IOPS. This is the mission-critical workload representing a Tier 1 application. Workload 2 (orange line) has a steady state IOPS consumption of 3300 IOPS, a Max Limit of 4500, a Min Goal at 1700 and a normal priority. Workload 3 (green line) has a steady state of 1500 IOPS and a QoS rule defining a Max Limit of 2000 IOPS, a Min Goal of 200 IOPS, and a low priority. The system in use is an HPE 3PAR StoreServ 7200 with a total IOPS capacity of around 9000. Operating at around 4000+3300+1500 = 8800 IOPS means it is near its maximum IOPS capacity. Because of the latency goal for Workload 1, the System Busy level will manage the resource distribution of the system in case of IOPS pressure.

By looking at the upper graph in Figure 20, you can see that at the 7-minute mark, Workload 1 bursts into a higher rate of IOPS consumption because of an activity. Because this brings the system at its 9000 IOPS limit around minute 10, the System Busy level algorithm inside Priority Optimization forces low-priority Workload 3 to give up I/O capacity in favor of Workload 1. IOPS in Workload 3 will be reduced until it reaches its Min Goal of 200 IOPS at around minute 13. After Workload 3 reaches its floor IOPS value, normal priority Workload 2 will be reduced in IOPS until Workload 1 reaches its new steady state of around 6800 IOPS. When Workload 1 drops off again around minute 36, the freed IOPS go to Workload 2 first, because it has a higher priority than Workload 3. With Workload 2 again in steady state at around minute 43, the rest of the freed IOPS go to Workload 3 bringing it in steady state again.

The second graph plots the service times for each of the three workloads throughout the same scenario. The service time of the Tier 1 workload (purple line) is marginally impacted by the increase of its increased IOPS requirement because the QoS subsystem pulled array resources from less critical applications. As a result, the service times of the normal (orange line) and low priority (green line) workloads go up. When the Tier 1 workload drops again to its initial steady-state level, service times are restored, first for the normal priority workload, followed by the low priority workload.

This example shows that mission-critical workloads can burst into higher IOPS or bandwidth by loaning capacity from other less critical workloads. The choice of where the required capacity comes from is controlled by the storage administrator determining the priority levels of the other workloads.


Overprovisioning performance limits Priority Optimization allows overprovisioning performance limits when creating QoS rules. Overprovisioning in this context means defining QoS rules with a combined value for the Max Limit for IOPS or bandwidth that exceeds the capability of the HPE 3PAR StoreServ system. This is an established practice if the workloads are sufficiently orthogonal in time for their I/O usage. Take the example of an HPE 3PAR StoreServ system running a database that is only queried during the day and a backup application that issues I/O requests only during the night. On the surface, no QoS rule seems to be needed in this scenario. However, you might want to consider setting up a dual QoS rule for each workload. A first rule, active during the workload’s scheduled hours of operation, sends out the entire I/O capacity of the system to that workload. A second rule limits the I/O capacity of that workload outside of these hours. This QoS setup prevents the backup workload from affecting the daytime database workload if it is accidentally or deliberately started during the day. The HPE 3PAR OS Task Scheduler switches automatically between the QoS rules for a workload during different periods of the day.

Priority Optimization with virtual domains HPE 3PAR StoreServ systems are well suited for multitenant environments inside an organization or at a service provider. Following best practices, a storage administrator creates a virtual domain per tenant and places the tenant’s VVs and hosts inside it. The administrator then creates a QoS rule for each domain to confine its I/O capacity to what is needed or what has been paid for. This QoS rule for the entire domain is called a “top-level” rule and governs I/O to all current and future objects attributed to the tenant.

Priority Optimization supports nesting QoS rules. This allows for the creation of secondary QoS rules on individual VVsets or groups of them in the domain of the tenant. These nested rules limit I/O for smaller objects within a tenant’s domain. Nested QoS rules can be overprovisioned without the risk of adversely affecting the workloads of other tenants because the top-level QoS rule enforces a Max Limit on the I/O capacity of all objects within the tenant’s domain.

Figure 21 shows a numerical example of top-level and nested QoS rules following these principles. The example shows a system with two tenants that were logically isolated from each other by creating their resources (VVs and hosts) inside a virtual domain. For each domain, top-level QoS rules called “Tenant-1” and “Tenant-2” were created. Max Limits for IOPS of respectively 30,000 and 70,000 were assigned to these two rules. The combined IOPS value given to these two tenants is equal to the I/O capability of 100,000 IOPS for the array. Three applications within the domain of Tenant-1 have volumes grouped in a VVset each. I/O by each VVset is limited by a second-level, nested QoS rule. The combined Max Limits of 20,000, 15,000, and 12,000 IOPS for these nested QoS rules exceed the IOPS value of 30,000 specified in the top-level QoS rule for the tenant.

Each application of Tenant-1 can reach its defined Max Limit for IOPS when the other applications in the domain are not using all the resources they have been assigned. If all of Tenant-1’s applications become busy at the same time, the IOPS capacity consumed will never exceed the Max Limit of 30,000 of the top-level QoS domain rule. This top-level QoS rule prevents applications inside Tenant-1’s domain to affect the resources assigned to Tenant-2. Domain Tenant-2 was allocated 70,000 IOPS and will not be able to grow over this threshold. This means it will not affect Tenant-1’s domain in case of local congestion.

Note that the use of domains in this example is efficient and secure but not required. VVsets can be nested as well and nested QoS rules on them are working as described in the example.


Figure 21. Top-level and nested QoS rules

QoS and Remote Copy With HPE 3PAR OS 3.1.3 and later, VVsets are automatically created for Remote Copy groups. These VVsets are assigned the prefix RCP_xxx, where xxx is the name of the Remote Copy group. Although such a VVset is accepted as an object for a QoS rule, the defined limits are not honored. This means QoS cannot control the pace of replication for Remote Copy.

When configuring a QoS policy with a latency goal against VVsets that are in synchronous replication mode, you should take into consideration the link latency and the write I/O service time on the secondary array. You can estimate the latency by adding the average round-trip time (RTT) of the Remote Copy link to 1.5X the average local write service time.

Influence of QoS on the host side When the IOPS or bandwidth demand for an application reaches the defined QoS limit, the array resources used by the application on the host no longer grow. Lowering the QoS limit for an application that has reached its defined limits generally frees up resources on the host, benefitting other workloads on that host. This means that Priority Optimization indirectly controls resource allocation on the host. Host-side and storage-side QoS rules can be combined to interact in a complementary manner for tighter control or when precise memory and CPU cycle consumption management is required on the host.

Maximum number of QoS rules per VV A VV can be part of a large number of VVsets. To reduce complexity and preserve performance predictability of applications, HPE does not encourage the use of many QoS rules applied to the same VV. Priority Optimization limits the number of QoS rules that can be applied to a VV to eight. The lowest value for the Max Limit of IOPS and bandwidth of any VVset containing the VV imposes its limits on the I/O traffic to and from the VV. In the same way the lowest Min Goal and smallest latency goal determine the behavior of the VV. When a volume is a member of two VVsets that have a different priority, the rule on the lowest-priority VVset takes precedence.


QoS on a subset of VVset volumes A QoS rule on a VVset governs all volumes in the set. The situation can arise where only a subset of the volumes in that VVset needs a QoS rule. To achieve this, create a secondary VVset out of the volumes in the top-level VVset that need a QoS rule and define the rule for the secondary level VVset. The secondary VVset does not need to be exported for the QoS rule to take action.

If the System rule is defined, it acts on all volumes and VVsets that do not have a QoS rule defined. If a VVset has no QoS rule defined on it but at least one volume of it is a member of another VVset that has a named QoS rule, that QoS rule takes precedence over the System rule, even if that rule has a lower Min Goal for IOPS or bandwidth.

QoS and snapshots Snapshots of volumes in a VVset do not belong automatically to the VVset. This means that the System rule and not the QoS rule for the VVset will govern snapshot performance. Because of a tight parent/child relationship for updating changed blocks, the application using the VVset volumes is impacted when the System rule restricts resource consumption to below the VVset QoS rule. HPE recommends adding snapshots to the VVset their parent volume belongs to.

Conclusion Priority Optimization software implements quality of service levels for applications and workloads as business requirements dictate. Mission-critical applications in enterprise environments are protected by assigning performance limits to workloads with lower service-level requirements. This brings predictability to the performance of applications and tenants in an HPE 3PAR StoreServ storage array.


Share now

Get updates

© Copyright 2019 Hewlett Packard Enterprise Development LP. The information contained herein is subject to change without notice. The only warranties for Hewlett Packard Enterprise products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. Hewlett Packard Enterprise shall not be liable for technical or editorial errors or omissions contained herein.

Microsoft and Windows Server are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. Oracle and Java are registered trademarks of Oracle and/or its affiliates. The OpenStack Word Mark is either a registered trademark/service mark or trademark/service mark of the OpenStack Foundation, in the United States and other countries and is used with the OpenStack Foundation’s permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation or the OpenStack community. Citrix is a registered trademark of Citrix Systems, Inc. and/or one more of its subsidiaries and may be registered in the United States Patent and Trademark Office and in other countries. VMware, VMware vSphere, and VMware ESXi are registered trademarks or trademarks of VMware, Inc. in the United States and/or other jurisdictions. All other third-party marks are property of their respective owners.

a00073435ENW, May 2019

Resources, contacts, or additional links HPE 3PAR StoreServ Management Console 3.4 User Guide https://support.hpe.com/hpsc/doc/public/display?docId=a00053601en_us

HPE Storage Consulting Services hpe.com/us/en/services/storage-services.html

HPE SPOCK hpe.com/storage/spock

HPE Pointnext hpe.com/us/en/services.html

Learn more at hpe.com/3par

http://www.hpe.com/info/getupdated

https://www.hpe.com/global/hpechat/index.html?jumpid=Collaterals_a00073435ENW

http://www.hpe.com/info/getupdated


https://www.hpe.com/us/en/services/storage-services.html

http://www.hpe.com/storage/spock

https://www.hpe.com/us/en/services.html

http://www.hpe.com/3par

HPE 3PAR Priority Optimization · HPE 3PAR Priority Optimization software for H PE 3PAR StoreServ...

Documents

Transcript of HPE 3PAR Priority Optimization · HPE 3PAR Priority Optimization software for H PE 3PAR StoreServ...