Introduction to Linux/Unix for HPCC users

XiaogeWang,ICERwangx147@msu.edu

Aug.16,2017

Goal •  Prerequisitefor“IntroducEontoHPCC”•  MostcommonlyusedbasicLinuxcommandsamongHPCCusers

•  UnderstandsimpleLinuxshellscript

How does this class work

Taskdrivenlearningmethod–  DescripEonofatask–  DemonstrateasoluEonofataskbyinstructor–  handsonexercises.

UsethesEckynotesprovidedtohelpmehelpyou.

–  Nos%cky=Iamworking–  Green=Iamdoneandreadytomoveon(yea!)–  Red=IamstuckandneedmoreEmeand/orsomehelp

Agenda

•  IntroducEon•  Linux

– Partone:•  ConnecttoHPCC•  Setupyourworkingenvironment•  Runyourprogram

– Parttwo:•  Scripts•  Otherusefulstuff

•  Summary

Agenda

•  Linux

– Partone:•  ConnecttoHPCC•  Setupyourworkingenvironment•  Runyourprogram

– Parttwo:•  Scripts•  Otherusefulstuff

•  Summary

Introduction

•  Bigmap–  Linux/Unix–  Shell

•  Overview– Commands–  SimpleShellscript– GetreadyforHPC.

Big picture

Shell Shell

Overview

•  Commands•  SimpleShellscript•  GetreadyforHPC

Agenda

•  IntroducEon

– Partone:•  ConnecttoHPCC•  Setupyourworkingenvironment•  Runyourprogram

– Parttwo:•  Scripts•  Otherusefulstuff

•  Summary

Agenda

•  IntroducEon•  Linux

•  ConnecttoHPCC•  Setupyourworkingenvironment•  Runyourprogram

– Parttwo:•  Scripts•  Otherusefulstuff

•  Summary

Agenda

•  IntroducEon•  Linux

– Partone:

•  Setupyourworkingenvironment•  Runyourprogram

– Parttwo:•  Scripts•  Otherusefulstuff

•  Summary

Task 1: Connect to HPCC •  Task:connectfromyourPCtoHPCCdevelopnodesusingyour

account.•  SoluEon:threesteps

–  Stepone:•  get/openyourclient

–  Terminalsession»  Windowsusers

MobaXterm»  Macusers

Terminal,»  Linuxusers

Terminal–  Remotedesktop:rdp.hpcc.msu.edu

–  Steptwo:•  $ssh–Xwangx147@hpcc.msu.edu,(rsync.hpcc.msu.edu)

–  Stepthree:•  $sshdev-nodename

•  Note:–  Firstcommand“ssh”,itisobenreferas“login”,“connect”–  Readmessageonloginpage.–  Howtoreportproblem:client,gateway,node,account,errormessage,

Eme,locaEon.

Connect to HPCC

•  Remotedesktop–  InstallRemoteDesktopclient– CreateaconnecEon

•  PCname:rdp.hpcc.msu.edu•  Username:•  Password:

–  Startremotedesktop•  Clickon“TerminalEmulator”

Exercise 1: connect to HPCC

•  Task:– Use/installyourclienttoconnecttooneofdevnode,makesureyoureadtheloginmessagesmessage

– Type“gmod”

Agenda

•  IntroducEon•  Linux

– Partone:•  ConnecttoHPCC

•  Runyourprogram– Parttwo:

•  Scripts•  Otherusefulstuff

•  Summary

Task 2: Set up working environment •  Gettoknowmyworkingplace

– Hardware–  Sobware–  Storage

•  Setupmyworkingenvironment– Obtainsobwarepackages– Preparedata–  Setenvironmentvariable

Task 2.1: Get info

1.  Systeminfo2.  Sobwareinfo3.  Storagespace

Task 2.1.1:System info

•  Systemandkernal$uname–a

•  Numberofcores$nproc–all

•  SizeofRam$free–h

•  CPUinfo$lscpu

Exercise 2.1.1: get system info

•  Getthesysteminfoofthedev-node:–  Systemandkernel–  Numberofcores–  SizeofRam(memory)–  CPUinfo

Task 2.1.2:Software info •  Viewthesystemavailablesobware

$modulelist#currentlyloaded$moduleavail#allavail$modulespider$moduleloadpowertools

•  Wherearetheyinstalled?$ls/opt/soCware

•  WhichversionamIusing?Numberofcores$whichpython$python--version

•  Isthereanyexamplesorhelpsofthesobwarepackages?$getexample$getexample<example_name>

Exercise 2.1.2:find a tool

(1)Checkifsobware“perbools”isavailable.Ifitisavailable,runfollowing“pprof”commandof“perbools”:

$pprof--helpDoyouknowwhereisitinstalled?(2)Findthelatestversionof“iw”andmoduleloaditfordevelopmentofsobware.Getexampleofiw.(3)FindtheversionofaCcompiler“cc”

Task 2.1.3:Storage info •  Whereismy“finder”or“windowsexplorer”?

–  Rootofsystem:/–  Homedirectory:~/,$HOME–  Researchdirectory(space):

/mnt/research/<name_group>–  Scratchspaceforuserandresearchgroup:

$SCRATCH,/mnt/scratch/wangx147–  /tmp(noprivatespace)

•  Size,usageandcontentofthespace?–  $quota,–  $df–h/mnt/research/helpdesk,–  $lfsquota-uwangx147/mnt/scratch–  $pwd,$cd<directory>,$ls,$ls–al,$ls–R

Exercise 2.1.3:get storage info

•  Findthesizeandcontentsofyour–  Homedirectory–  Scratchspace–  /tmp

•  Findtheusageofdirectory/opt/sobware

Navigation

•  Understandstructureofstorage–  Spaces(DiskparEEon)

•  home:/mnt/home/NetID•  Scratch:/mnt/scratch/NetID•  Research:/mnt/research/GroupName

–  Directories(folder)•  “tree”

–  Files•  Filename:absolute/relaEve

–  Shortcuts•  ~,$HOME,$SCRTACH,..,./

Task 2.2: Set environment •  Obtainsobwarepackages

–  SobwareinstallaEon(notcovered)– Moduleload(done)

•  Preparedata–  Aboutafile– Makeadirectoryforfiles:$mkdir–  TransferfilesfromotherlocaEon–  Createfilelocallyusinganeditor

•  Setenvironmentvariables$exportPATH=/mnt/home/wangx147/bin:$PATH$exportOMP_NUM_THREADS=4$printenv

All about a file

•  Aoributes:–  Accesspermission–  Owner–  Group–  Size– ModificaEonEme–  name

•  Example:$ls-lundelivered.err-rw-rw-r--1wangx147staff-np0Aug216:33undelivered.err

File Access Permission •  Example:class0@dev-intel14-k20~]$ls-ltotal66-rw-r--r--1class0temporary48672Sep52012cheatsheet.pdf-rwxr-xr-x1class0temporary660Jan12015Colorfull.shdrwxr-xr-x2class0temporary3Oct1217:06Documents^^^^^^^^^^|||||||||||||||ownergroupsizeEmename||||numberoflinkstodirectorycontent|||permissionforallusers||permissionforthemembersofthegroup|permissionforownerTypeoffile.-=file,d=directory,l=link•  Changepermission

$chmod

Ownership

•  Changeonwership– $chownowner:groupfilename– $chown:groupfilename

File Name

•  Fullname•  RelaEvename•  Changefilename:

$mvold_namenew_name

Task 2.2.1: Get or Create Files

•  Getfilesfromotherplaces–  $cpsource_filedest_file–  $wgetweb_address–  $curlweb_address– Moremethodswillbecoveredby“introtoHPCC”

•  Createfileslocally–  Createdirectory:mkdirdirname–  Redirectstandardoutputtofile:>,>>,>&,1>,2>–  Editor:nano

•  Note:

|| visible in terminal || visible in file || existing Syntax || StdOut | StdErr || StdOut | StdErr || file ==== ======++==========+==========++==========+==========++=========== > || no | yes || yes | no || overwrite >> || no | yes || yes | no || append2> || yes | no || no | yes || overwrite 2>> || yes | no || no | yes || append &> || no | no || yes | yes || overwrite &>> || no | no || yes | yes || append

Exercise 2.2.1: Create a file

(1)  getacheatsheetfromInternetaddhop://steve-parker.org/sh/cheatsheet.pdf;

(2)  createadirectory“workshop”thatcontainsit;

(3)  Copyallfilesin/opt/sobware/zlib/1.2.8/includeto“workshop”

(4)  Createafile“List”whichrecordedallthefilesinthedirectory.

Hint:$cp/tmp/*.Couldcopyallfilein/tmptocurrentdirectory

Task 2.2.2: Look into Files

•  Openatextfilewithoutclickingmouse–  cat,head,head–n,tail,tail–n,more,less

•  SearchforaparEcularpaoernfromfiles–  grep,grep–r,grep–I,grep-v

•  OtherproperEesofafile– wc,

Exercise 2.2.2: Closer look

•  Task:Viewthecontentsofthedatafile“polls.csv”.FindouthowmanypollswererecordedfromMichigan.– GototheDATAdirectoryandviewthefile– Use“MI”tofindthedatarecordedfromMichigan.

Task 2.2.3Search for Files

•  Searchafilewith– Name–  Size– Time– Type

•  Example:findallversionofperl$find/opt/sobware/perl-nameperl-execls-l{}+$find/opt/sobware/perl-nameperl|grepbin

Exercise 2.2.3: Searching files

•  Task:findouthowmanyversionsofgcccompilerininstalledin/opt/sobware/GCC.

•  Hint:gcccompilerisstoredin…/bin/directory.Not…/lib/directory

Agenda

•  IntroducEon•  Linux

– Partone:•  ConnecttoHPCC•  Setupyourworkingenvironment

– Parttwo:•  Scripts•  Otherusefulstuff

•  Summary

Task 3: run your program

•  Launchaprogramlocallyondevelopnodewithoutclickingmouse–  “cd”toworkinglocaEon–  SpecifyInputfiles–  SpecifyOutputfiles–  Environmentvariables

•  Launchbatchjobstorunoncomputenodes–  qsub(nocoveredhere)

•  Example$matlab–nodisplay–rtest_control_system

Monitor your program

•  $ps•  $top

LETS TAKE A BREAK

Agenda

•  IntroducEon•  Linux

– Partone:•  ConnecttoHPCC•  Setupyourworkingenvironment•  Runyourprogram

•  Otherusefulstuff•  Summary

From Command to Script

Shell script

CombinemorecommandstogetherformoresophisEcate/complicatedtask.•  Variables

–  Environment:$HOME,$HOSTNAME,$PATH,$SCRATCH,$PWD,

–  Internallydefined–  DefinefromCommandlinearguments

•  expressions,expansion•  Controlflow(pipeline,branch,loops)•  ExecuEon(sourcevs.directrun,)

Control Flow: Pipeline •  Connectseveralcommandsinsequence,usestandardoutputofacommandasstandardinputofnextcommand

•  Example:countnumberoffilesunder/bin.–  SoluEon1:

ls/bin>bin_listwcbin_list

–  SoluEon2:ls/bin|wc

•  Learned:–  Ifastandardoutputcouldbeusedasstandard

inputofnextcommand,apipelinecouldbebuild–  Commonlyusedcommandsforpipelining:

•  grep,wc,sort,less,more,uniq,head,tail,…

Examples of pipeline

•  Filteringoutputshowq|grepBatchHold

•  SorEngtheoutputls–l|sort

•  CounEngls|wc-l

•  Beoerviewingps|more

Exercise 4: Pipeline

•  Task:Findouthowmanyjobsinjobqueue.Amongthem,howmanyarerunning.

1.  Counttotaljobsinjobqueue.Use“showq”togetlistofjobsinqueue.

2.  Countonlytherunningjobs.Usekeyword“Running”tofilterthejobs

Control Flow: branch

•  ControltheworkflowaccordingtocertaincondiEon

•  Formatif[condiEonistrue];then…dosomething...else…dosomethingelsehere...fi

Example :Branch

•  Task:Writeascript“scratch2home”thatwilltakeafilenameasinputandbackupthefilefromscratchspacetohomespace.Copyisneededonlyifthefileisneweronscratch.

•  CondiEon:fileisnewer•  Learn:

– ExpressionofcondiEon– Branch

Exercise 5: Branch

•  Task:followexampletowriteascript“home2scratch”thatwilltakeafilenameasinputandcopythefilefromhomespacetoscratchspaceifthefileisneweronhomespace.

Control Flow: Loop •  Runsomethingrepeatedlyover•  Format

–  Forloop:#!/bin/bashforiin$(ls);doechoitem:$idone

–  Whileloop#!/bin/bashCOUNTER=0while[$COUNTER-lt10];doechoThecounteris$COUNTERletCOUNTER=COUNTER+1done

–  UnElloop#!/bin/bashCOUNTER=20unEl[$COUNTER-lt10];doechoCOUNTER$COUNTERletCOUNTER-=1done

Example 6 :Loop

•  Task:SimilarasExample5exceptthatittakeadirectorynameasinputandbackupawholedirectory.Writeascript“scratch2home_dir”

•  Loopcount–  StaEc/dynamic

•  Learn:–  Loop

Exercise 6: Loop

•  Task:SimilarasExercise9exceptthatittakesadirectorynameasinputandcopyfilesinthedirectoryfromHomespacetoscratchspaceifanyfilesarenewer.Writeascript“home2scratch_dir”todoit.

•  Followexample10

Make Your Own Command

•  Makeshellscript“executable”– Chmod+x

•  Beawareof“environment”•  ExecuEon

–  “sourcename_script”vs.“./name_script”

Example 6 :Start Project •  Task:Getreadyforworkonaprojectonscratch.Createa

commandcalled“start_project”.Itwilldothefollowing–  Copyfilestoworkspace–  Updateaworklogfile–  Makesureitisready:datafiles–  Makethewholetaskasacommand

•  Commands–  $cp–rBuildModel$SCRATCH–  $date>>$SCRATCH/BuildModel/log–  $ls-l$SCRATCH/BuildModel/

•  Learn:–  Ifataskcouldbeimplementedbymorethansingleline

command/pipelineandisneededrepeatedly–  Environmentvariable$SCRATCH–  >>vs.>–  Filepermissionneedtosettoexecutable–  Additto$PATH

Exercise 6: Finish Project

•  Task:finishupaprojectworkingonscratch.– Runonscratchworkingdirectory– Backupdirectory– gobacktohomedirectory

•  followexample11tomakeacommand“finish_project”

Agenda

•  IntroducEon•  Linux

– Partone:•  ConnecttoHPCC•  Setupyourworkingenvironment•  Runyourprogram

– Parttwo:•  Scripts•  Otherusefulstuff

Agenda

•  IntroducEon•  Linux

– Partone:•  ConnecttoHPCC•  Setupyourworkingenvironment•  Runyourprogram

– Parttwo:•  Scripts

•  Summary

Self Learning

•  Task:findoutwhatisthecommandtoshufflelinesofadatafile“pet_store.csv”,thencreateafile“pet_shuffled.csv”.

•  Commands– man,-help,man-k–  Googlesearch“howtoshufflelinesinafilelinuxshell”,“randompermutaEoninlinuxshell”

•  Learn:–  Howtofindcommand–  Howtogetdetailsofacommand–  Command“shuf”

Exercise 7: Seft Learning

•  Task:findouthowtosortlinesofadatafile,thensortthefile“polls.csv”to“polls_sort.csv”.1.  findoutifthereisacommandcouldsort

thefile.2.  Trytosortdatafile“polls.csv”.

Note:Donotsortthefirstline!

Summary •  Linuxlearnt

–  Commands•  NavigaEon•  Getorcreatefiles•  Organizingfiles•  Lookintofiles•  Searchfiles•  Fileaoributes•  Onlinehelpormanual

–  Scripts•  Pipeline•  Makeyouowncommand•  Environmentofashell

Q & A

Thank You!