Minnesota Supercomputing Institute · © 2013 Regents of the University of Minnesota. All rights...
Transcript of Minnesota Supercomputing Institute · © 2013 Regents of the University of Minnesota. All rights...
![Page 1: Minnesota Supercomputing Institute · © 2013 Regents of the University of Minnesota. All rights reserved. Introduction to Job Submission and Scheduling Andrew Gustafson](https://reader036.fdocuments.in/reader036/viewer/2022062605/5fd25502a5ac0f44f317415d/html5/thumbnails/1.jpg)
© 2013 Regents of the University of Minnesota. All rights reserved.
Minnesota SupercomputingInstitute
![Page 2: Minnesota Supercomputing Institute · © 2013 Regents of the University of Minnesota. All rights reserved. Introduction to Job Submission and Scheduling Andrew Gustafson](https://reader036.fdocuments.in/reader036/viewer/2022062605/5fd25502a5ac0f44f317415d/html5/thumbnails/2.jpg)
© 2013 Regents of the University of Minnesota. All rights reserved.
Introduction to Job Submission and Scheduling
Andrew Gustafson
![Page 3: Minnesota Supercomputing Institute · © 2013 Regents of the University of Minnesota. All rights reserved. Introduction to Job Submission and Scheduling Andrew Gustafson](https://reader036.fdocuments.in/reader036/viewer/2022062605/5fd25502a5ac0f44f317415d/html5/thumbnails/3.jpg)
© 2013 Regents of the University of Minnesota. All rights reserved.
Interacting with MSI Systems
![Page 4: Minnesota Supercomputing Institute · © 2013 Regents of the University of Minnesota. All rights reserved. Introduction to Job Submission and Scheduling Andrew Gustafson](https://reader036.fdocuments.in/reader036/viewer/2022062605/5fd25502a5ac0f44f317415d/html5/thumbnails/4.jpg)
© 2013 Regents of the University of Minnesota. All rights reserved.
Connecting to MSISSH is the most reliable connection method
Linux and Mac users can use the terminal command:ssh login.msi.umn.edu
Windows users will need to use an ssh capable program, like PuTTY or Cygwin.
SSH connections must first connect to login.msi.umn.edu. From there you can connect to other systems.
For graphical connections use NICE:www.nice.msi.umn.edu
![Page 5: Minnesota Supercomputing Institute · © 2013 Regents of the University of Minnesota. All rights reserved. Introduction to Job Submission and Scheduling Andrew Gustafson](https://reader036.fdocuments.in/reader036/viewer/2022062605/5fd25502a5ac0f44f317415d/html5/thumbnails/5.jpg)
© 2013 Regents of the University of Minnesota. All rights reserved.
MSI Computing Environment
MSI systems are primarily Linux compute clusters running CentOS.
Software is managed via a module system.
Jobs are scheduled via a queueing system.
Home directories are unified across systems.
![Page 6: Minnesota Supercomputing Institute · © 2013 Regents of the University of Minnesota. All rights reserved. Introduction to Job Submission and Scheduling Andrew Gustafson](https://reader036.fdocuments.in/reader036/viewer/2022062605/5fd25502a5ac0f44f317415d/html5/thumbnails/6.jpg)
© 2013 Regents of the University of Minnesota. All rights reserved.
Machine Architecture: Cluster
Source: http://en.wikipedia.org/wiki/Cluster_%28computing%29
![Page 7: Minnesota Supercomputing Institute · © 2013 Regents of the University of Minnesota. All rights reserved. Introduction to Job Submission and Scheduling Andrew Gustafson](https://reader036.fdocuments.in/reader036/viewer/2022062605/5fd25502a5ac0f44f317415d/html5/thumbnails/7.jpg)
© 2013 Regents of the University of Minnesota. All rights reserved.
○ Mesabi■ About 17,700 total cores, on Intel Haswell processors.■ 24 cores and 62gb per node in the large primary queues.■ Special queues with large memory (up to 1TB), and GPUs.■ Allows node sharing: good for both small and large jobs.■ mesabi.msi.umn.edu
○ Itasca■ About 9,000 total cores, on Intel Nehalem processors.■ 8 cores and 22gb per node in the large primary queue.■ Special queues with larger memory and 16 cores per node.■ itasca.msi.umn.edu
○ Lab Server■ About 500 total cores, on older hardware.■ For interactive, or small single node jobs.■ 8 cores and 15gb per node in the primary queue.■ lab.msi.umn.edu
Clusters at MSI
![Page 8: Minnesota Supercomputing Institute · © 2013 Regents of the University of Minnesota. All rights reserved. Introduction to Job Submission and Scheduling Andrew Gustafson](https://reader036.fdocuments.in/reader036/viewer/2022062605/5fd25502a5ac0f44f317415d/html5/thumbnails/8.jpg)
© 2013 Regents of the University of Minnesota. All rights reserved.
Clusters at MSI
Mesabi LabItasca
Login
First connect to login.msi.umn.edu,then connect to a cluster.
Must be on-campus, or using the VPN.https://it.umn.edu/virtual-private-network-vpn
![Page 9: Minnesota Supercomputing Institute · © 2013 Regents of the University of Minnesota. All rights reserved. Introduction to Job Submission and Scheduling Andrew Gustafson](https://reader036.fdocuments.in/reader036/viewer/2022062605/5fd25502a5ac0f44f317415d/html5/thumbnails/9.jpg)
© 2013 Regents of the University of Minnesota. All rights reserved.
Home Directories
Home directories are unified across all Linux systems.
Each group has a disk quota which can be viewed with the command: groupquota
Panasas ActivStor 14: 3.01PB storage, capable of 30 GB/sec read/write, and 270,000 IOPS
![Page 10: Minnesota Supercomputing Institute · © 2013 Regents of the University of Minnesota. All rights reserved. Introduction to Job Submission and Scheduling Andrew Gustafson](https://reader036.fdocuments.in/reader036/viewer/2022062605/5fd25502a5ac0f44f317415d/html5/thumbnails/10.jpg)
© 2013 Regents of the University of Minnesota. All rights reserved.
Loading SoftwareSoftware modules are used to alter environmental variables, in order to make software available. MSI has hundreds of software modules.
Description Command Example
See all available modules: module avail module avail
Load a module: module load module load matlab/2015a
Unload a module: module unload module unload matlab/2015a
Unload all modules: module purge module purge
See what a module does: module show module show matlab/2015a
List currently loaded modules: module list module list
Module Commands:
![Page 11: Minnesota Supercomputing Institute · © 2013 Regents of the University of Minnesota. All rights reserved. Introduction to Job Submission and Scheduling Andrew Gustafson](https://reader036.fdocuments.in/reader036/viewer/2022062605/5fd25502a5ac0f44f317415d/html5/thumbnails/11.jpg)
© 2013 Regents of the University of Minnesota. All rights reserved.
Job SchedulingOn MSI systems, calculations are performed within “jobs”. A job is a planned calculation that will run for a specified time length on a specified set of hardware.
There are two types of job:1. Non-interactive (vast majority)2. Interactive
The job scheduler front-end is called the Portable Batch System (PBS).
Jobs start in your home directory with no modules loaded.
![Page 12: Minnesota Supercomputing Institute · © 2013 Regents of the University of Minnesota. All rights reserved. Introduction to Job Submission and Scheduling Andrew Gustafson](https://reader036.fdocuments.in/reader036/viewer/2022062605/5fd25502a5ac0f44f317415d/html5/thumbnails/12.jpg)
© 2013 Regents of the University of Minnesota. All rights reserved.
Job ScriptsTo submit a non-interactive job, first make a
PBS job script.
Example:#!/bin/bash -l#PBS -l walltime=8:00:00,nodes=3:ppn=8,pmem=1000mb#PBS -m abe#PBS -M [email protected]
cd ~/program_directorymodule load intelmodule load ompi/intelmpirun -np 24 program_name < inputfile > outputfile
![Page 13: Minnesota Supercomputing Institute · © 2013 Regents of the University of Minnesota. All rights reserved. Introduction to Job Submission and Scheduling Andrew Gustafson](https://reader036.fdocuments.in/reader036/viewer/2022062605/5fd25502a5ac0f44f317415d/html5/thumbnails/13.jpg)
© 2013 Regents of the University of Minnesota. All rights reserved.
Job SubmissionTo submit a job script use the command:
qsub -q queuename scriptname
A list of queues available on different systems can be found here:https://www.msi.umn.edu/queues
Submit jobs to a queue which is appropriate for the resources needed.
Resources to consider when choosing a queue:● Walltime● Total cores and cores per node● Memory● Special hardware (GPUs, etc)
![Page 14: Minnesota Supercomputing Institute · © 2013 Regents of the University of Minnesota. All rights reserved. Introduction to Job Submission and Scheduling Andrew Gustafson](https://reader036.fdocuments.in/reader036/viewer/2022062605/5fd25502a5ac0f44f317415d/html5/thumbnails/14.jpg)
© 2013 Regents of the University of Minnesota. All rights reserved.
Job SubmissionTo view queued jobs use the commands:
qstat -u usernameshowq -w user=username
For detailed information:checkjob -v jobnumber
To cancel a submitted job use the command:qdel jobnumber
![Page 15: Minnesota Supercomputing Institute · © 2013 Regents of the University of Minnesota. All rights reserved. Introduction to Job Submission and Scheduling Andrew Gustafson](https://reader036.fdocuments.in/reader036/viewer/2022062605/5fd25502a5ac0f44f317415d/html5/thumbnails/15.jpg)
© 2013 Regents of the University of Minnesota. All rights reserved.
Interactive Jobs
Nodes may be requested for interactive use using the command:
qsub -I -X -l walltime=1:00:00,nodes=1:ppn=8,mem=2gb
The job waits in the queue like all jobs, and when it begins the terminal returns control.
![Page 16: Minnesota Supercomputing Institute · © 2013 Regents of the University of Minnesota. All rights reserved. Introduction to Job Submission and Scheduling Andrew Gustafson](https://reader036.fdocuments.in/reader036/viewer/2022062605/5fd25502a5ac0f44f317415d/html5/thumbnails/16.jpg)
© 2013 Regents of the University of Minnesota. All rights reserved.
Service Units (SUs)
Jobs on the high performance computing (HPC) systems consume Service Units (SUs), which roughly correspond to processor time.
Each research group is given a service unit allocation at the beginning of the year. To view the number of service units remaining use the command: acctinfo
If a group is using service units faster than the "fairshare target", then the group's jobs will have lower queue priority.
![Page 17: Minnesota Supercomputing Institute · © 2013 Regents of the University of Minnesota. All rights reserved. Introduction to Job Submission and Scheduling Andrew Gustafson](https://reader036.fdocuments.in/reader036/viewer/2022062605/5fd25502a5ac0f44f317415d/html5/thumbnails/17.jpg)
© 2013 Regents of the University of Minnesota. All rights reserved.
Simple Parallelization: BackgroundingMost easily done with single node jobs.
#!/bin/bash -l#PBS -l walltime=8:00:00,nodes=1:ppn=8,pmem=1000mb#PBS -m abe#PBS -M [email protected]
cd ~/job_directorymodule load example/1.0./program1.exe < input1 > output1 &./program2.exe < input2 > output2 &./program3.exe < input3 > output3 &./program4.exe < input4 > output4 &./program5.exe < input5 > output5 &./program6.exe < input6 > output6 &./program7.exe < input7 > output7 &./program8.exe < input8 > output8 &wait
![Page 18: Minnesota Supercomputing Institute · © 2013 Regents of the University of Minnesota. All rights reserved. Introduction to Job Submission and Scheduling Andrew Gustafson](https://reader036.fdocuments.in/reader036/viewer/2022062605/5fd25502a5ac0f44f317415d/html5/thumbnails/18.jpg)
© 2013 Regents of the University of Minnesota. All rights reserved.
Simple Parallelization: Job ArraysWorks best on Mesabi.
Template Job Script, template.pbs:#!/bin/bash -l#PBS -l walltime=8:00:00,nodes=1:ppn=8,pmem=1000mb#PBS -m abe#PBS -M [email protected]
cd ~/job_directorymodule load example/1.0./program.exe < input$PBS_ARRAYID > output$PBS_ARRAYID
Submit an array of 10 jobs:qsub -t 1-10 template.pbs
![Page 19: Minnesota Supercomputing Institute · © 2013 Regents of the University of Minnesota. All rights reserved. Introduction to Job Submission and Scheduling Andrew Gustafson](https://reader036.fdocuments.in/reader036/viewer/2022062605/5fd25502a5ac0f44f317415d/html5/thumbnails/19.jpg)
© 2013 Regents of the University of Minnesota. All rights reserved.
Minnesota Supercomputing Institute
The University of Minnesota is an equal opportunity educator and employer. This PowerPoint is available in alternative formats upon request. Direct requests to Minnesota Supercomputing Institute, 599 Walter library, 117 Pleasant St. SE,
Minneapolis, Minnesota, 55455, 612-624-0528.
Web: www.msi.umn.edu
Email: [email protected]
Telephone: (612) 626-0802