Adaptive Computing: Multi Parameter Fair-Share using ... · Multi Parameter Fair-Share using...

© 2015 ADAPTIVE COMPUTING, INC. 1

Adaptive Computing: Multi Parameter Fair-Share

using exponential and linear algorithms

ZKI Workshop

Bernhard Schott

Adaptive Computing EMEA

Oct 2016

© 2015 ADAPTIVE COMPUTING, INC. 2 © 2014 ADAPTIVE COMPUTING, INC. 2

Agenda

▪ Multi Parameter Fair-Share using exponential and linear algorithms:

▪ Moab overview

▪ Overview Fair-Share methods

▪ Fair-Share exponential and linear usage tracking

▪ Fair-Share enforcement configuration options

▪ Moab Accounting Manager (MAM): Fund-Fair-Share

▪ Fair-Share-Scheduling & Backfill

▪ Questions and Answers


Adaptive Computing

Oil and Gas, Financial, Manufacturing, Research, Academic and Government

Largest provider of HPC job/workload management software Sources: IDC HPC End-user Study of System Software and Middleware in Technical Computing, 2013. Intersect360 Research 2012 HPC Site Census

Pioneer & innovator in scheduling and optimization

80+ patents issued or pending applicable to world-class HPC, cloud and big data solutions

Global partnerships include SGI, Intel, HP, Lenovo, Cray, & Microsoft

Managing Leading Fortune 500 and Top 500 Systems


Moab Intelligence Engine

Core Moab® Technology

▪ 15+ years battle tested

▪ Patented (80+ Patents)

▪ Mimics real-world decision-making

Multi-dimensional Policies Optimize Across:

▪ Workload/application requirements

▪ Priorities and SLAs

▪ Time (real-time and future, predictive)

▪ Heterogeneous resources


Intelligent Workload Optimization

Moab® HPC Suite - Enterprise Edition

Storage Provisioning Queue Mgr

(Torque/Other) Network

Health Monitor

Non-traditional

CLI or 3rd Party End user Portal

Admin Dashboard

External System Report Interface

Resource Managers

HW Resources

Other Resources: Licenses Disk Etc.

Workload


Decisions, policies, scheduling, allocation, & orchestration

Accounting

APIs (Web Services, CLI, etc.)


Moab structural overview: single logical cluster


Showback & Chargeback

Web Services API

Self Service Portal

Admin Dashboard

External System Reports

Resource Mgr TORQUE

Cluster 1

The resource manager manages the nodes, launches and controls the workload. Torque can include resources from multiple physical “groups of resources”.

All compute nodes would belong to one “partition” and could be managed and used transparently as one single system image.

Scheduler

Ext. Interface Layer

Resource Manager

Cluster 2


Moab structural overview: Multi-Cluster

Resources can be of different type: Big SMP, clusters, new architectures ... and adding Nitro to perform HTC

Scheduler


Resource Manager

Self Service Portal

Admin Dashboard




Web Services API

Big SMPs

Resource Mgr Torque

Cluster

NITRO

Resource Mgr Torque

cluster


Moab structural overview: Multiple Clusters

Resources can be of different type: big SMP, compute node clusters, … ... and grouped into multiple clusters or groups

Scheduler


Resource Manager

Self Service Portal

Admin Dashboard




Web Services API

Cluster 2 Cluster 1

Resource Manager Torque

Cluster 3

Moab provides a single system image to the user


Moab & multiple resource manager partitions

Optionally, multiple clusters or groups of resources can be controlled by individual resource managers – perfect to integrate clusters of non-routed compute nodes

Scheduler


Resource Manager

Moab provides a single system image to the user

Self Service Portal

Admin Dashboard




Web Services API

Cluster 2 Cluster 1

Resource Mgr SLURM

Cluster 3

Resource Mgr Torque 2

Resource Mgr Torque 3


Moab structural overview: Grid

Large scale reference use case: www.hlrn.de


Web Services API

Resource Mgr TORQUE_A


Web Services API

Resource Mgr TORQUE_B Resources from all sites are

available to all users, controlled by policies. Can be combined with the previously described configurations.

Portal / CLI Portal / CLI Two or more sites, peer or hierarchical configurations


Moab structural overview: add NITRO



Web Services API

Self Service Portal

Admin Dashboard


Resource Mgr TORQUE

Each NITRO session invokes a transient subcluster with tremendous performance. Invoke as many NITRO sessions as needed.

Test results: 10 million tasks executed in 12 minutes => 13500 tasks/s done in a single NITRO session. (submission rate is much higher)

Scheduler


Resource Manager

NITRO NITRO


Ease of Use Experience

Dmarsh – dmarsh@ac-moab256:/tmp 230x60

End User CLI or Submission & Application Portal

Administrator Dashboard / Portal

Management Accounting / Usage Reporting

• CLI or App Portal • Job Templates • Fast Commands

• Powerful Search • GUI Job Mgmt. • Visual Timelines

• Flexible Charging • Detailed Tracking • Historical Reports


Fair-Share-Scheduling overview


Fairness: Full-term Resource Balancing

Balance usage & sharing of resources across stakeholders and avoid last minute rushes.

Long Term Mid Term Short Term Months to Quarters

Budgets/Allocations (MAM)

Hours to Weeks Fairshare

Instantaneous Usage Limits

1st Allocation

Period

2nd Allocation

Period

3rd Allocation

Period

4th Allocation

Period

Plan & allocate according to org. objectives • Lump Sum • Stepped / Inverted Steps • Overlapping – Staggered

Balance mid-term surges and avoid late usage • Adds priority to under-

utilized accounts • Reduces priority to over-

utilized accounts

Control Current Usage • Hard Limits • Soft Limits (Impact priority) • Overrides • Reservations (Time specific

or long standing)

User 1 User 2 User 3 User 4

Not Enough Just Right Too Much

Override

Hard Limit

Soft Limit

6 5 4 3 2 1


Fair-Share-Scheduling usage tracking

Linear & Exponential


Fairshare

Moab Fair-Share

Fairshare is configured at two levels. First, at a system level, configuration is required to determine how fairshare usage information is to be collected and processed.


Fairshare: system level and credential level

The following are system level parameters:

Second, some configuration is required at the credential level to determine how this fairshare information affects particular jobs. Credential level configuration consists of specifying fairshare utilization targets using the *CFG suite of parameters, including ACCOUNTCFG, CLASSCFG, GROUPCFG, QOSCFG, and USERCFG.

http://docs.adaptivecomputing.com/9-0-2/MWM/Content/topics/moabWorkloadManager/topics/appendices/a.aparameters.html#accountcfg

http://docs.adaptivecomputing.com/9-0-2/MWM/Content/topics/moabWorkloadManager/topics/appendices/a.aparameters.html#classcfg

http://docs.adaptivecomputing.com/9-0-2/MWM/Content/topics/moabWorkloadManager/topics/appendices/a.aparameters.html#groupcfg

http://docs.adaptivecomputing.com/9-0-2/MWM/Content/topics/moabWorkloadManager/topics/appendices/a.aparameters.html#qoscfg

http://docs.adaptivecomputing.com/9-0-2/MWM/Content/topics/moabWorkloadManager/topics/appendices/a.aparameters.html#usercfg


FSPOLICY - Specifying Metric of Consumption

FSPOLICY parameter allows selection of both the types of resources to be tracked as well as the method of tracking.

Example for FSPOLICY :a 4-processor job is running a parallel /bin/sleep for 15 minutes. It will have a dedicated fairshare usage of 1 processor-hour but a consumed fairshare usage of essentially nothing since it did not consume anything.

FSPOLICY DEDICATEDPS% FSINTERVAL 24:00:00 FSDEPTH 28 FSDECAY 0.75

http://docs.adaptivecomputing.com/9-0-2/MWM/Content/topics/moabWorkloadManager/topics/appendices/a.aparameters.html#fspolicy


Fairshare: Specifying Fairshare Decay

Many sites want to limit the impact of fairshare data according to its age. The FSDECAY parameter allows this, causing the most recent fairshare data to contribute more to a credential's total fairshare usage than older data. The table shows the impact of a number of commonly used decay factors on the percentage contribution of each fairshare window.



FSDECAY parameter allows exponential decay for previous use records.

Graph: Blue: decay f 1,00 Red: decay f 0,80 Green: decay f 0,75 Purple: decay f 0,50

Decay factor 0 1 2 3 4 5 6 7

1,00 100% 100% 100% 100% 100% 100% 100% 100%

0,80 100% 80% 64% 51% 41% 33% 26% 21%

0,75 100% 75% 56% 42% 32% 24% 18% 13%

0,50 100% 50% 25% 13% 6% 3% 2% 1%

Fair-Share Windows



Decay factor 0 1 2 3 4 5 6 7 8 9 10 11 12

1,00 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100%

0,80 100% 80% 64% 51% 41% 33% 26% 21% 17% 13% 11% 9% 7%

0,75 100% 75% 56% 42% 32% 24% 18% 13% 10% 8% 6% 4% 3%

0,50 100% 50% 25% 13% 6% 3% 2% 1% 0% 0% 0% 0% 0%

Fair-Share WindowsFSDECAY parameter allows exponential decay for previous use records. In case of decay factor 1 previous use is “remembered” completely (100%) until end of Fair-Share time frame = linear behavior.

Note: decay factor 0,50 renders pointless from FS-Window 8 onwards.




FSDECAY parameter 1,00 or close to 1,00 create (almost) linear decay of previous use records.

Choice of decay factor depends on the wished “remembering” of previous use. Decay factor 1,00 (=linear) would result in not forgetting anything until end of fairshare time frame.

Decay factor 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

1,00 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100%

0,99 100% 99% 98% 97% 96% 95% 94% 93% 92% 91% 90% 90% 89% 88% 87% 86% 85% 84% 83% 83% 82% 81% 80% 79% 79% 78% 77% 76% 75% 75% 74% 73%

0,98 100% 98% 96% 94% 92% 90% 89% 87% 85% 83% 82% 80% 78% 77% 75% 74% 72% 71% 70% 68% 67% 65% 64% 63% 62% 60% 59% 58% 57% 56% 55% 53%

0,95 100% 95% 90% 86% 81% 77% 74% 70% 66% 63% 60% 57% 54% 51% 49% 46% 44% 42% 40% 38% 36% 34% 32% 31% 29% 28% 26% 25% 24% 23% 21% 20%

Fair-Share Windows



Fair-Share enforcement: user & workload type

specific, soft limits & hard limits


Fairshare: Fair-Share Trees

Configuration is required at the credential level to determine how fairshare information affects particular jobs. Here some examples:

Fair-Share Trees

Moab supports arbitrary depth (no hard limit) hierarchical Fair-Share-Scheduling based on a share tree. In this model, users, groups, classes, and accounts can be arbitrarily organized and their usage tracked and limited. Moab extends common share tree concepts to allow mixing of credential types, enforcement of ceiling and floor style usage targets, and mixing of hierarchical Fair-Share-Scheduling state with other priority components. Hierarchical Fair-Share trees are configured in XML.

FSTREE[myTree]

<fstree>

<tnode name="root" share="100">

<tnode name="john" type="user" share="50" limits="MAXJOB=8 MAXPROC=24 MAXWC=01:00:00">

</tnode>

<tnode name="jane" type="user" share="50" limits="MAXJOB=5">

</tnode>

</tnode>

</fstree>


Fairshare: Hierarchical Fair-Share Trees

Hierarchical Fair-Share Cont.

This configuration creates an example Fair-Share-Tree in which 2 users share a value of 100. Users John and Jane share the value equally, because each has been given 50. In addition, optional limits are defined for both users. Limits could be: MAXIJOB, MAXJOB, MAXMEM, MAXNODE, MAXPROC, MAXSUBMITJOBS, MAXWC.

Note: comparing with the Backfill discussion, we understand that Backfill utilization target may be hindered by these Fair-Share limits. On the other hand, no single user could flood the cluster.

FSTREE[myTree]

<fstree>

<tnode name="root" share="100">

<tnode name="john" type="user" share="50" limits="MAXJOB=8 MAXPROC=24 MAXWC=01:00:00">

</tnode>

<tnode name="jane" type="user" share="50" limits="MAXJOB=5">

</tnode>

</tnode>

</fstree>


Fairshare: multiple partitions

Hierarchical Fair-Share for multiple Partitons

FSTREE[tree] <fstree>

<tnode partition=“part1" name="root" type="acct" share="100" limits="MAXJOB=6"> <tnode name="accta" type="acct" share="50" limits="MAXSUBMITJOBS=2 MAXJOB=1"> <tnode name="fred" type="user" share="1" limits="MAXWC=1:00:00"> </tnode> </tnode> <tnode name="acctb" type="acct" share="50" limits="MAXSUBMITJOBS=4 MAXJOB=3"> <tnode name="george" type="user" share="1" > </tnode> </tnode> </tnode>

<tnode partition=“part2" name="root" type="acct" share="100" limits="MAXSUBMITJOBS=6 MAXJOB=5"> <tnode name="accta" type="acct" share="50"> <tnode name="paul" type="user" share="1"> </tnode> </tnode> <tnode name="acctb" type="acct" share="50"> <tnode name="ringo" type="user" share="1"> </tnode> </tnode> </tnode> </fstree>


Fairshare: dynamically updated

Dynamically Importing Fair-Share-Tree Data:

Share trees can be centrally defined and this information can be dynamically imported and used within Moab.

Example: save a fair-share tree xml file as e.g. fstree.dat and link it to Moab using the IDCFG parameter in your Moab configuration file.

Moab imports the myTree fairshare tree from the fstree.dat file. Setting REFRESHPERIOD to INFINITY causes Moab to read the file each time it starts or restarts, but setting a positive interval (e.g. 4:00:00) cause Moab to read the file more often. See Refreshing Identity Manager Data for more information.

IDCFG[myTree] server="FILE:///$MOABHOMEDIR/etc/fstree.dat" REFRESHPERIOD=INFINITY

http://docs.adaptivecomputing.com/9-0-2/MWM/Content/topics/moabWorkloadManager/topics/appendices/a.aparameters.html#idcfg

http://docs.adaptivecomputing.com/9-0-2/MWM/Content/topics/moabWorkloadManager/topics/misc/idmanager.html#refresh




Fairshare: adjustable impact per partition

▪ Moab uses the FSSCALINGFACTOR attribute for QOS credentials to get the calculated fairshare usage of a job.

▪ Moab will multiple the actual fairshare usage by this value to get the calculated fairshare usage of a job. The actual fairshare usage is calculated based on the FSPOLICY parameter.

▪ For an example, if FSPOLICY is set to DEDICATEDPS and a job runs on two processors for 100 seconds then the actual fairshare usage would be 200.

▪ If the job ran on a qos with FSSCALINGFACTOR=.5 then Moab would multiply 200*.5=100.

▪ If the job ran on a partition with FSSCALINGFACTOR=2 then Moab would multiply 200*2=400.

QOSCFG[qos1] FSSCALINGFACTOR=<double>



Fairshare: targets & caps

Fairshare Targets: Fairshare targets can be applied to user, group, account, QoS, or class credentials using the FSTARGET attribute of *CFG credential parameters. These targets allow fairshare information to affect job priority and each target can be independently selected.

Target type - Ceiling

Target modifier -

Job impact Priority

Format Percentage Usage

Description

Adjusts job priority down when usage exceeds target. See How violated ceilings and floors affect fairshare-based priority for more information on how ceilings affect job priority.

Target type - Floor

Target modifier +

Job impact Priority


Description

Adjusts job priority up when usage falls below target. See How violated ceilings and floors affect fairshare-based priority for more information on how floors affect job priority.

Target type - Target

Target modifier N/A

Job impact Priority


Description Adjusts job priority when usage does not meet target.

http://docs.adaptivecomputing.com/9-0-2/MWM/Content/topics/moabWorkloadManager/topics/prio_res/priorityfactors.html#ceilingsFloors



























Fairshare Targets: Example

▪ The following example increases the priority of jobs belonging to user john until he reaches 16.5% of total cluster usage. All other users have priority adjusted both up and down to bring them to their target usage of 10%:

FSPOLICY DEDICATEDPS FSWEIGHT 1 FSUSERWEIGHT 100 USERCFG[john] FSTARGET=16.5+ USERCFG[DEFAULT] FSTARGET=10 ...



Fairshare Caps

Caps can be applied to users, accounts, groups, classes, and QoSs using the FSCAP attribute of *CFG credential parameters. Unlike fairshare targets, if a credential reaches its fairshare cap, its jobs can no longer run and are thus removed from the eligible queue and placed in the blocked queue. In this respect, fairshare targets behave like soft limits and fairshare caps behave like hard limits.

Absolute Cap

Cap Modifier: ^

Job Impact: Feasibility

Format: Absolute Usage

Description: Constrains job eligibility as an absolute quantity measured according to the scheduler charge metric as defined by the FSPOLICY parameter

Relative Cap

Cap Modifier: %

Job Impact: Feasibility

Format: Percentage Usage

Description:

Constrains job eligibility as a percentage of total delivered cycles measured according to the scheduler charge metric as defined by the FSPOLICY parameter.
















Fairshare Caps: Example

The following example constrains the marketing account to use no more than 16,500 processor seconds during any given floating one week window. At the same time, all other accounts are constrained to use no more than 10% of the total delivered processor seconds during any given one week window.

FSPOLICY DEDICATEDPS FSINTERVAL 12:00:00 FSDEPTH 14 ACCOUNTCFG[marketing] FSCAP=16500^ ACCOUNTCFG[DEFAULT] FSCAP=10 ...


Fairshare: mdiag -f

View Share Tree

Fair-Share-Tree configuration can be viewed with mdiag -f. The tree information will appear beneath all the information about applied Fair-Share settings configured in moab.cfg.

> mdiag -f

Share Tree Overview for partition 'ALL'

Name Usage Target (FSFACTOR)

---- ----- ------ ------------

root 100.00 100.00 of 100.00 (node: 1171.81) (0.00)

- john 16.44 50.00 of 100.00 (user: 192.65) (302.04) MAXJOB=8 MAXPROC=24 MAXWC=3600

- jane 83.56 50.00 of 100.00 (user: 979.16) (-302.04) MAXJOB=5


Moab Accounting Manager (MAM)

Fund-Fair-Share


▪ Sets usage budget amounts for users, groups, accounts

▪ Resource allocations and scheduling checked against amounts

▪ Tracks and reports system usage by each user, group, account, etc. for showback or pay-for-use chargeback

▪ Allows for rich resource costing

Moab Usage Accounting

20%

20% 15%

25%

20%

SLA

Job Request

Moab

Checks budget

Allocates & schedules

NODES

Set Usage Budgets

Track usage & determine charges

Budget & usage reports for showback or chargeback

Engineering A

Finance

Physics Chemistry

Life-Science

$

$

$

$ $

updates budget


Accounting Mode in MAM

● strict-allocation – Use this mode if you wish to strictly

enforce allocation limits. Under this mode, jobs can be

prevented from running if the end-users do not have sufficient

funds. This is the default.

● fast-allocation – Use this mode if you wish to debit

allocations, but need higher throughput by eliminating the lien

and quote of strict-allocation mode.

● notional-charging – Use this mode if you wish to calculate

and record charges for workload usage, but not keep track of

fund balances or allocation limits.

● usage-tracking – Use this mode if you wish to record

workload usage details, but not to calculate a charge nor keep

track of fund balances or allocation limits


Fair-Share-Scheduling & MAM

Funds

Consumption

Funds

Consumption

Fair-Share-Scheduling Moab Accounting Manager

Days - Weeks Quarterly to semi-annual to annual

REFIL

L

Intra-day and day-to-day Fairness

Long term Fairness

&


Fair-Share-Scheduling & MAM

Funds

Consumption

Funds

Consumption

Fair-Share-Scheduling Moab Accounting Manager

Days - Weeks Quarterly to semi-annual to annual

REFIL

L

Intra-day and day-to-day Fairness

Long term Fairness

& Fair-Share impacts job priority Fund usage has no

impact on job priority


Fair-Share-Scheduling & Backfill


▪ Workload packing and continuous scheduling optimization

▪ Considers all job requests in queue in priority order

▪ Fully uses all available capacity to maximize throughput (backfill)

▪ Provides up to 20% utilization performance improvement

Optimized, Intelligent Scheduling

Time

* Job Queue

*

*

Nodes

Future Reservation

Reservation

Main

tenance

Reserv

ation

P1

P7

P8 P2

P14

P11

P4 *

P5 *

P3

P9

P10

P15

P16

P6 *

90-99% utilization

P13

P1

P16 P12

NOW

Reservation

Reservation


Resource Job Timeline

• Viewpoint portal provided view of the executed scheduling plan.

• Jobs presented in boxes with different sizes and shades to indicate

• Number of cores

• Size (darker the box the larger the job)


Liberal versus Conservative Backfill

By default, Moab reserves only the highest priority job resulting in a liberal and aggressive backfill. This reservation guarantees that backfilled jobs will not delay the highest priority job, although they may delay other jobs. The parameter RESERVATIONDEPTH controls how conservative or liberal the backfill policy is. This parameter controls how deep down the queue priority reservations will be made. While decreasing this parameter improves guarantees that priority jobs will not be bypassed, it reduces the freedom of the scheduler to backfill resulting in somewhat lower system utilization. The significance of the trade-offs should be evaluated on a site by site basis.

Backfilled although low in priority

http://docs.adaptivecomputing.com/9-0-2/MWM/Content/Resources/Graphics/backfill.gif

http://docs.adaptivecomputing.com/9-0-2/MWM/Content/topics/moabWorkloadManager/topics/appendices/a.aparameters.html#reservationdepth








Summary


Moab enforced Fairness

o Moab Fair-Share-Scheduling provides o Dynamically enforced fairshare Targets

o Relative and absolute fairshare Caps

o Arbitrarily deep structured fairshare Trees

o Linear and exponential decay of previous use records

o Moab Accounting Manager (MAM) enforces budgets & funds based fairness

o Time scales o Fair-Share-Scheduling is perfect for intra-day to a few weeks

o MAM is perfectly suited to control any long term intervals: months, quarters, years.

o Together with soft and hard limits, all three methods provide fairness and governance over all time scales.


Moab enforced Fairness

Questions?

o Contact us, we are happy to discuss and find the best possible design for your HPC or HTC system! o [email protected]

o [email protected]

mailto:[email protected]



Adaptive Computing: Multi Parameter Fair-Share using ... · Multi Parameter Fair-Share using...

Documents

Transcript of Adaptive Computing: Multi Parameter Fair-Share using ... · Multi Parameter Fair-Share using...