Building a SLURM Banking System - Research...

24
Building a SLURM Banking System Authors Sahil Hasan*, Harrison Kuo*, Cassie Zhang*, Quinn Dombrowski*, Steve Masover*, Patrick Schmitz*, Krishna Muriki(*,**) Yong Qin(*,**) * Berkeley Research Computing, University of California, Berkeley * High Performance Computing Services, Lawrence Berkeley National Laboratory

Transcript of Building a SLURM Banking System - Research...

Page 1: Building a SLURM Banking System - Research ITresearch-it.berkeley.edu/sites/default/files/SLURM-Users-Group...minutes and tracks burndown of sub-allocations within an account Components.

BuildingaSLURMBankingSystemAuthorsSahilHasan*,HarrisonKuo*,CassieZhang*,QuinnDombrowski*,SteveMasover*,PatrickSchmitz*,KrishnaMuriki(*,**)YongQin(*,**)

*BerkeleyResearchComputing,UniversityofCalifornia,Berkeley*HighPerformanceComputingServices,LawrenceBerkeleyNationalLaboratory

Page 2: Building a SLURM Banking System - Research ITresearch-it.berkeley.edu/sites/default/files/SLURM-Users-Group...minutes and tracks burndown of sub-allocations within an account Components.

myBRCDashboardUIDatabaseSystemRESTAPISLURMPlugins

- Overview-

BuildingaSLURMBankingSystem

Page 3: Building a SLURM Banking System - Research ITresearch-it.berkeley.edu/sites/default/files/SLURM-Users-Group...minutes and tracks burndown of sub-allocations within an account Components.

• BerkeleyResearchComputing

• Deliverreliable,sustainableresourcesandservicestomeetcomputationaldemandsofResearchgroupsinUCBerkeley.

• Savioisa400nodeinstitutionalHPCLinuxclusterwhichformsthefoundationoftheBRCprogram.

• NumberofresearchgroupswithCondocontributions-22

• Numberofresearchgroupsusingfreecomputeallowances- 195

• Numberofuniqueusers- 1120

WhatisBRC?

Page 4: Building a SLURM Banking System - Research ITresearch-it.berkeley.edu/sites/default/files/SLURM-Users-Group...minutes and tracks burndown of sub-allocations within an account Components.

• Managedthroughsacct,sacctmgr

• CurrentBRCaccountinginfrastructureislimitedandnoteasilyintegratedintomyBRC

• HardtocreatewebapplicationstoprovideuserinterfacesforaccountingusingSLURMAPI

StateofSLURMAccounting

Page 5: Building a SLURM Banking System - Research ITresearch-it.berkeley.edu/sites/default/files/SLURM-Users-Group...minutes and tracks burndown of sub-allocations within an account Components.

Why?

• Facultygetcomputeallowanceof300KServiceUnits(SU=1core-hour)

• FacultyComputingAllowances(FCA)areusuallyformedforfacultyresearch•FCA~10softrustedusersinagroup

• InstructionComputingAllowances(ICA)areusuallygivenforinstructors•ICA~10-100sofpotentiallyuntrustedusers

• Instructors/Facultywanttocontrolquotasandourcurrentinfrastructurecannotsupportthis

Page 6: Building a SLURM Banking System - Research ITresearch-it.berkeley.edu/sites/default/files/SLURM-Users-Group...minutes and tracks burndown of sub-allocations within an account Components.

Goals

• Createanopensource,plugandplayaccountingsystem

• IndependentofexistingSLURMdatabase

• Easilyallowforwebapplicationstobebuiltontopofstack

Page 7: Building a SLURM Banking System - Research ITresearch-it.berkeley.edu/sites/default/files/SLURM-Users-Group...minutes and tracks burndown of sub-allocations within an account Components.

Overview

SLURMplugins

API+Logic

SLURMplugins myBRCDashboard

Database

Page 8: Building a SLURM Banking System - Research ITresearch-it.berkeley.edu/sites/default/files/SLURM-Users-Group...minutes and tracks burndown of sub-allocations within an account Components.

• AgraphicalDjango-baseddashboardthataidsaccountownersinchangingallocations,generatingvisualizations,etc.

• AdatabasesystemseparatefromexistingSLURMinfrastructure

• RESTAPItoprotectintegrityofdatabase

• Asystemofpluginsthatenableseasysub-allocationofTRES-minutesandtracksburndownofsub-allocationswithinanaccount

Components

Page 9: Building a SLURM Banking System - Research ITresearch-it.berkeley.edu/sites/default/files/SLURM-Users-Group...minutes and tracks burndown of sub-allocations within an account Components.

DatabaseSystemRESTAPISLURMPlugins

Overview

- myBRCDashboardUI-

BuildingaSLURMBankingSystem

Page 10: Building a SLURM Banking System - Research ITresearch-it.berkeley.edu/sites/default/files/SLURM-Users-Group...minutes and tracks burndown of sub-allocations within an account Components.

DashboardIntegratedwithBRCWebsite

Page 11: Building a SLURM Banking System - Research ITresearch-it.berkeley.edu/sites/default/files/SLURM-Users-Group...minutes and tracks burndown of sub-allocations within an account Components.

Login

Page 12: Building a SLURM Banking System - Research ITresearch-it.berkeley.edu/sites/default/files/SLURM-Users-Group...minutes and tracks burndown of sub-allocations within an account Components.

NotificationCenter

Page 13: Building a SLURM Banking System - Research ITresearch-it.berkeley.edu/sites/default/files/SLURM-Users-Group...minutes and tracks burndown of sub-allocations within an account Components.

StatisticsVisualization

Page 14: Building a SLURM Banking System - Research ITresearch-it.berkeley.edu/sites/default/files/SLURM-Users-Group...minutes and tracks burndown of sub-allocations within an account Components.

AllocationManager

Page 15: Building a SLURM Banking System - Research ITresearch-it.berkeley.edu/sites/default/files/SLURM-Users-Group...minutes and tracks burndown of sub-allocations within an account Components.

ServiceUnitCalculator

Page 16: Building a SLURM Banking System - Research ITresearch-it.berkeley.edu/sites/default/files/SLURM-Users-Group...minutes and tracks burndown of sub-allocations within an account Components.

RESTAPISLURMPlugins

OverviewmyBRC DashboardUI

- DatabaseSystem-

BuildingaSLURMBankingSystem

Page 17: Building a SLURM Banking System - Research ITresearch-it.berkeley.edu/sites/default/files/SLURM-Users-Group...minutes and tracks burndown of sub-allocations within an account Components.

DatabaseSystem

• NeededtoreflectwhatwasusedinSLURMaccounting

• Neededtohavegoodassociationsbetweenusers,accountsandjobs

• Customizablepartitions,QOS,jobstatus,etc.

• UsesmySQL

Page 18: Building a SLURM Banking System - Research ITresearch-it.berkeley.edu/sites/default/files/SLURM-Users-Group...minutes and tracks burndown of sub-allocations within an account Components.
Page 19: Building a SLURM Banking System - Research ITresearch-it.berkeley.edu/sites/default/files/SLURM-Users-Group...minutes and tracks burndown of sub-allocations within an account Components.

SLURMPlugin

OverviewmyBRC DashboardUIDatabaseSystem

- RESTAPI-

BuildingaSLURMBankingSystem

Page 20: Building a SLURM Banking System - Research ITresearch-it.berkeley.edu/sites/default/files/SLURM-Users-Group...minutes and tracks burndown of sub-allocations within an account Components.

RestAPI

• Neededtosupportviewingquotas

• Neededtoallowprivilegeduserstoeditotherusers’quotas

• Threeendpoints:/users,/accounts,and/jobs

• Allowsalayerofconvenienceandsecuritybetweentheuser/applicationandthedatabase/logic/SLURM

• UsesDjangoWebFramework

Page 21: Building a SLURM Banking System - Research ITresearch-it.berkeley.edu/sites/default/files/SLURM-Users-Group...minutes and tracks burndown of sub-allocations within an account Components.

- SLURMPlugin-

OverviewmyBRC DashboardUIDatabaseSystemRESTAPI

BuildingaSLURMBankingSystem

Page 22: Building a SLURM Banking System - Research ITresearch-it.berkeley.edu/sites/default/files/SLURM-Users-Group...minutes and tracks burndown of sub-allocations within an account Components.

SLURMPlugin

• SLURMpluginsperformwebAPIcallstointeractwithdatabase

• Allowsforjobsrunfromtheterminaltoberegisteredindatabase

• Splitintotwofunctions:asubmit-timeandanepilogue

Page 23: Building a SLURM Banking System - Research ITresearch-it.berkeley.edu/sites/default/files/SLURM-Users-Group...minutes and tracks burndown of sub-allocations within an account Components.

LifecycleofaSuccessfulJob

Job submittedBy user

Job created with hold status

Submit-Time

Job updatedwith finish

status Check SUs used

Epilogue

Check if job is less than allowance

Deducts user balance

Job run

Page 24: Building a SLURM Banking System - Research ITresearch-it.berkeley.edu/sites/default/files/SLURM-Users-Group...minutes and tracks burndown of sub-allocations within an account Components.

Thankyou


Introduction to Slurm - NASA

Introduction to Slurm - NASA

Slurm Job Scheduling - hprc.tamu.edu

Slurm Job Scheduling - hprc.tamu.edu

Tutorial 3: Cgroups Support On SLURM · Slurm User Group 2012 (c) Bull 2012 Tutorial 3: Cgroups Support On SLURM SLURM User Group 2012, ... SLURM Cgroups Documentation • Cgroups

Tutorial 3: Cgroups Support On SLURM · Slurm User Group 2012 (c) Bull 2012 Tutorial 3: Cgroups Support On SLURM SLURM User Group 2012, ... SLURM Cgroups Documentation • Cgroups

Slurm at UPPMAX - Uppsala University

Slurm at UPPMAX - Uppsala University

SLURM Deployment Experiences on Stampede

SLURM Deployment Experiences on Stampede

Advanced UPPMAX usage More SLURM

Advanced UPPMAX usage More SLURM

Evaluating SLURM simulator with real-machine SLURM and ...

Evaluating SLURM simulator with real-machine SLURM and ...

IGDPD C27 Burndown Chart Template

IGDPD C27 Burndown Chart Template

Bright Cluster Manager & SLURM - SchedMD :: SLURM Development

Bright Cluster Manager & SLURM - SchedMD :: SLURM Development

Slurm Workload Manager - Open MPI

Slurm Workload Manager - Open MPI

CRUK cluster practical sessions (SLURM)

CRUK cluster practical sessions (SLURM)

Introduction to Abel and SLURM - Universitetet i oslo · Introduction to Abel and SLURM Katerina Michalickova ... . …

Introduction to Abel and SLURM - Universitetet i oslo · Introduction to Abel and SLURM Katerina Michalickova ... . …

Slurm Migration

Slurm Migration

Resource management with Slurm

Resource management with Slurm

RESEARCH COMPUTING PBS SLURM TRANSITION

RESEARCH COMPUTING PBS SLURM TRANSITION

EMBL Slurm - EMBL Bio-IT Portal

EMBL Slurm - EMBL Bio-IT Portal

SLURM At CSCS - SchedMD · June 2010—First working port of SLURM to a Cray ... Fall 2010March 2011 Initial experimentation with the use of SLURM ... SigTerm vs SigKill changes to

SLURM At CSCS - SchedMD · June 2010—First working port of SLURM to a Cray ... Fall 2010March 2011 Initial experimentation with the use of SLURM ... SigTerm vs SigKill changes to

Slurm Solutions - adaptivecomputing.com

Slurm Solutions - adaptivecomputing.com

Overview of Slurm version 2.6

Overview of Slurm version 2.6

Bright Cluster Manager - SchedMD :: SLURM Development

Bright Cluster Manager - SchedMD :: SLURM Development