Workshop: High-performance computing for economists · Stampede (no. 6 on Top500 as of June 2013)...

Intro Basics VCS HPC resources

Workshop: High-performance computing foreconomists

Lars Vilhuber1 John M. Abowd1 Richard Mansfield1

Hautahi Kingi1 Flavio Stanchi1 Sida Peng1

Kevin L. McKinney

1Cornell University, Economics Department,

August 17-19, 2015

Vilhuber, Abowd, Mansfield, McKinney Computing for Economists

Workshop: High-performance computing for economists

Back in the days...

RAM: 2,000 words (2kB); Speed: 2 MHzSource: Wikipedia

Back in the days...

RAM: 2,000 words (2kB); Speed: 2 MHzSource: Wikipedia

They went to the moon

Source: Flickr

Big progress

RAM: 2 ×32 kB; Speed: 1 MHz, $1,500 (today’s USD)Wikipedia

RAM: 2 ×10242 kB; Speed: 1.700 MHz × 4$700 (today’s USD) Source: Wikipedia

We still fly to the moon

Source CNET

This is where you can go

Stampede (no. 6 on Top500 as of June 2013)

RAM: 192 ×10243 kB, Speed: 2,700 Mhz × 462,462Source: TACC

This is where you can go

Stampede (no. 6 on Top500 as of June 2013)

RAM: 192 ×10243 kB, Speed: 2,700 Mhz × 462,462Source: TACC

But first...

http://viewfromwitsend.wordpress.com/

But first...

http://viewfromwitsend.wordpress.com/

What do you learn in a Ph.D. program?

How to learn...

What do you learn in a Ph.D. program?How to learn...

Goal of this class

To open new doors, to be able to conceive of problems that youdidn’t think had a feasible solution.To broaden your knowledge about what you do NOT know

Goal of this classTo open new doors, to be able to conceive of problems that youdidn’t think had a feasible solution.

To broaden your knowledge about what you do NOT know

Goal of this classTo open new doors, to be able to conceive of problems that youdidn’t think had a feasible solution.To broaden your knowledge about what you do NOT know

So in order to do that...

Overview

I Programming basics (Lars)

I Choosing an editorI How to structure programs, texts, etc.I A clean sequence of programsI NX, SSH, Linux, request an account on clusterI Basic scripting

I Basics of version control (Lars)

I File-system based version controlI More formal version control (Subversion, Git)I Working with serversI Setting up infrastructure at Cornell

I HP resources at Cornell, elsewhere

Overview

I Programming basics (Lars)I Choosing an editor

I How to structure programs, texts, etc.I A clean sequence of programsI NX, SSH, Linux, request an account on clusterI Basic scripting

Overview

I Programming basics (Lars)I Choosing an editorI How to structure programs, texts, etc.

I A clean sequence of programsI NX, SSH, Linux, request an account on clusterI Basic scripting

Overview

I Programming basics (Lars)I Choosing an editorI How to structure programs, texts, etc.I A clean sequence of programs

I NX, SSH, Linux, request an account on clusterI Basic scripting

Overview

I Programming basics (Lars)I Choosing an editorI How to structure programs, texts, etc.I A clean sequence of programsI NX, SSH, Linux, request an account on cluster

I Basic scriptingI Basics of version control (Lars)

Overview

I Programming basics (Lars)I Choosing an editorI How to structure programs, texts, etc.I A clean sequence of programsI NX, SSH, Linux, request an account on clusterI Basic scripting

Overview

I Basics of version control (Lars)I File-system based version control

I More formal version control (Subversion, Git)I Working with serversI Setting up infrastructure at Cornell

Overview

I Basics of version control (Lars)I File-system based version controlI More formal version control (Subversion, Git)

I Working with serversI Setting up infrastructure at Cornell

Overview

I Basics of version control (Lars)I File-system based version controlI More formal version control (Subversion, Git)I Working with servers

I Setting up infrastructure at CornellI HP resources at Cornell, elsewhere

Overview

I Basics of version control (Lars)I File-system based version controlI More formal version control (Subversion, Git)I Working with serversI Setting up infrastructure at Cornell

Overview

I Basics of version control (Lars)I File-system based version controlI More formal version control (Subversion, Git)I Working with serversI Setting up infrastructure at Cornell

Overview

Structure of the class

Teaching...We’ll take you on a 4,000 m flight through topics...

... and practice

... and then swoop in on some examples, leaving ample time topractice it.

Structure of the class

Teaching...We’ll take you on a 4,000 m flight through topics...

... and practice

... and then swoop in on some examples, leaving ample time topractice it.

Choosing editors

Why does choosing editors matter?The (applied) research process iterates through writing papersand doing estimation. You want to use the appropriate tools foreach task.

Integrated or separate

I You can use native tools that come with each wordprocessing facility/programming language/etc.

I Not all of them will have one.I Not all of them will work on all platforms.I You will likely use multiple tools

Choosing an editor

... or systemSeparate editors and systems

I MS Word and math editor(Windows/OSX butcompatibility issues)

I LibreOffice(Windows/OSX/Linux butnot as good)

I NotePad++ (Windows)I Gedit, (X)Emacs, Kate

(Linux)I Sublime Text (OSX)I Atom (all, see also MS

Visual Studio Code)

LATEX: all platforms, but someGUIs are not cross-platform,ease of use varies:

I TeXstudio (all platforms)I TeXMaker (all platforms)I Scientific Workplace

(Windows, mythicalLinux)

I TeXWorks+MiktexI TEXnicCenterI and (many more)

Choosing an editor

... or systemIntegrating programming andrunning

I IDE ( Eclipse, ActiveStateKomodo, etc.)

I Native programming GUIs(SAS, Matlab, Stata)

I Gedit, (X)Emacs (withadd-on functionality)

Integrating programs andtext/results

I SWeave/knitr (integratesLATEX and R)

I RStudio (GUI to R andSWeave/knitr)

I Shiny (web interface to Rwith dynamic results)

I StatRep (Integrated SASand LATEX, Source 1,Source 2)

Structuring programs

Easy...

Listing 1: mystuff.sas1 data ”C:\Users\Me\CensusChina . sas7bdat ” ;2 set ”C:\Users\Me\CensusChina . sas7bdat ” ;3 earn= log ( earn ) ;4 run ;5 proc reg data =”C:\Users\Me\CensusChina . sas7bdat ” ;6 model earn = sex educat ion exper ience ;7 run ;

What can possibly be wrong about that?

Structuring programs 2

Easier...

Listing 2: mystuff.do1 use ”C:\Users\Me\CensusChina . dta ”2 rep lace earn= log ( earn )3 regress earn sex educat ion exper ience4 save , rep lace

What can possibly be wrong about that?

Actually...Everything!

I Name of program: uninformativeI Destruction of original data: program cannot be re-run for

same resultsI No portability: cannot be run anywhere elseI No explanation: why are we doing this?

But of course, nobody does that, right?

Better...?

Listing 3: china-regression.sas1 data logCensusChina ;2 set ”C:\Users\Me\CensusChina . sas7bdat ” ;3 earn= log ( earn ) ;4 run ;5 proc reg data=logCensusChina ;6 model earn = sex educat ion exper ience ;7 run ;

Somewhat...

Better...?

Listing 4: china-regression.sas1 data logCensusChina ;2 set ”C:\Users\Me\CensusChina . sas7bdat ” ;3 earn= log ( earn ) ;4 run ;5 proc reg data=logCensusChina ;6 model earn = sex educat ion exper ience ;7 run ;

Somewhat...

Addressing these issues

I Naming of programs: hereI Commenting: hereI Versioning: up nextI Portability and Data management: tomorrow

Key notions about naming

Think of yourself as highly amnesiac...

I The research paper you are writing now will be submitted,rejected, worked on, questioned...

I ... by others and yourselfI ... in intervals of weeks, months, years...I Your future research assistant and the future YOU will

need to understand how to go through it.

I ... by others and yourself

I ... in intervals of weeks, months, years...I Your future research assistant and the future YOU will

I ... by others and yourselfI ... in intervals of weeks, months, years...

I Your future research assistant and the future YOU willneed to understand how to go through it.

I ... by others and yourselfI ... in intervals of weeks, months, years...I Your future research assistant and the future YOU will

Naming

The really bad

mystu f f .Rread .Rvers ion2 .Ro ls . sas

The bad

readCensus .RreadBLS .RprepareCensus .RrunOLS . sas

Naming

The really bad

mystu f f .Rread .Rvers ion2 .Ro ls . sas

The bad

readCensus .RreadBLS .RprepareCensus .RrunOLS . sas

Naming

Better

01 readBLS .R02 readCensus .R03 prepareCensus .R04 c r e a t e a n a l y s i s d a t a .R05 runOLS . sas

Even better

01 01 readBLS .R02 01 readCensus .R02 02 prepareCensus .R03 0 1 c r e a t e a n a l y s i s d a t a .R04 01 runOLS . sasREADME. t x t

Naming

Better

01 readBLS .R02 readCensus .R03 prepareCensus .R04 c r e a t e a n a l y s i s d a t a .R05 runOLS . sas

Even better

01 01 readBLS .R02 01 readCensus .R02 02 prepareCensus .R03 0 1 c r e a t e a n a l y s i s d a t a .R04 01 runOLS . sasREADME. t x t

Naming

Going overboard?i c f / c t r l p r o g s / c o n t r o l i c f . sasi c f / c t r l p r o g s / pa ramete rs i c f . sasi c f / l i b r a r y / macros / i c f c l e a n u p . sasi c f / l i b r a r y / macros / i c f i m p u t e c o u n t y r e s . sasi c f / l i b r a r y / macros / l i c f f i n d n u m . sasi c f / l i b r a r y / macros / l i c f p r o x y . sasi c f / l i b r a r y / macros / l i c f s t a r s 1 . sasi c f / l i b r a r y / macros / l i c f t g r l a t l o n g s . sasi c f / l i b r a r y / sasprogs /01 i c f q a . sasi c f / l i b r a r y / sasprogs /01 i c f . sasi c f / l i b r a r y / sasprogs /02 i c f q a . sasi c f / l i b r a r y / sasprogs /02 i c f . sasi c f / l i b r a r y / sasprogs /03 i c f q a . sasi c f / l i b r a r y / sasprogs /03 i c f . sas[ sn ip ]i c f / l i b r a r y / sasprogs /19 i c f . sas

ehf / c t r l p r o g s / c o n t r o l e h f . sasehf / l i b r a r y / macros / read b ls . sasehf / l i b r a r y / sasprogs /01 eh f . sas[ sn ip ]

Naming

Going overboard?i c f / c t r l p r o g s / c o n t r o l i c f . sasi c f / c t r l p r o g s / pa ramete rs i c f . sasi c f / l i b r a r y / macros / i c f c l e a n u p . sasi c f / l i b r a r y / macros / i c f i m p u t e c o u n t y r e s . sasi c f / l i b r a r y / macros / l i c f f i n d n u m . sasi c f / l i b r a r y / macros / l i c f p r o x y . sasi c f / l i b r a r y / macros / l i c f s t a r s 1 . sasi c f / l i b r a r y / macros / l i c f t g r l a t l o n g s . sasi c f / l i b r a r y / sasprogs /01 i c f q a . sasi c f / l i b r a r y / sasprogs /01 i c f . sasi c f / l i b r a r y / sasprogs /02 i c f q a . sasi c f / l i b r a r y / sasprogs /02 i c f . sasi c f / l i b r a r y / sasprogs /03 i c f q a . sasi c f / l i b r a r y / sasprogs /03 i c f . sas[ sn ip ]i c f / l i b r a r y / sasprogs /19 i c f . sas

ehf / c t r l p r o g s / c o n t r o l e h f . sasehf / l i b r a r y / macros / read b ls . sasehf / l i b r a r y / sasprogs /01 eh f . sas[ sn ip ]

Naming

With minor modificationi c f / c t r l p r o g s / c o n t r o l i c f . sasi c f / c t r l p r o g s / pa ramete rs i c f . sasi c f / l i b r a r y / macros / i c f c l e a n u p . sasi c f / l i b r a r y / macros / i c f i m p u t e c o u n t y r e s . sasi c f / l i b r a r y / macros / l i c f f i n d n u m . sasi c f / l i b r a r y / macros / l i c f p r o x y . sasi c f / l i b r a r y / macros / l i c f s t a r s 1 . sasi c f / l i b r a r y / macros / l i c f t g r l a t l o n g s . sasi c f / l i b r a r y / sasprogs /01 i c f . sasi c f / l i b r a r y / sasprogs /02 i c f . sasi c f / l i b r a r y / sasprogs /03 i c f . sas[ sn ip ]i c f / l i b r a r y / sasprogs /19 i c f . sasi c f / l i b r a r y / sasprogs /01 i c f q a . sasi c f / l i b r a r y / sasprogs /02 i c f q a . sasi c f / l i b r a r y / sasprogs /03 i c f q a . sas

Can you figure out in what sequence to run them?

Why SSH?

Most compute clusters have ONLY SSH accessIt is thus worthwhile to learn enough about it here, in order tobe functional there: CAC “Red Cloud”, Amazon Cloud, XSEDE.

Linux rules... the HPC worldAll 10 of the top 10 TOP500 computers run Linux (as thecompiler front-end, if not compute OS)

Graphical access

Two types of graphical access

I with an “X server” (native in Linux, optional in Windowsand OSX)

→ standard way on most clustersI using NX client software for improved experience

Graphical access

I with an “X server” (native in Linux, optional in Windowsand OSX)→ standard way on most clusters

I using NX client software for improved experience

Graphical access

I with an “X server” (native in Linux, optional in Windowsand OSX)→ standard way on most clusters

I using NX client software for improved experience

Basic Linux, basic scripting

Why worry?

You will end up doing something on the command line

I Launch a program from a compute-cluster job

I Launch a job submissionI Basic scripting

Why worry?

I Launch a program from a compute-cluster jobI Launch a job submission

I Basic scripting

Why worry?

I Launch a program from a compute-cluster jobI Launch a job submissionI Basic scripting

Linux in 2 minutes

I ls - will list the contents of a directoryI cd - will “change directory”I cd .. (note the spaces) will go up a directoryI cd (name) will go into the directory (name)I rm (name) will deleteI mkdir (name) will create a directory called (name)I vi (name) will open a venerable command line editor for file

(name)

(CAUTION: to exit, hit ESC, then :q!)

Linux in 2 minutes

I ls - will list the contents of a directoryI cd - will “change directory”I cd .. (note the spaces) will go up a directoryI cd (name) will go into the directory (name)I rm (name) will deleteI mkdir (name) will create a directory called (name)I vi (name) will open a venerable command line editor for file

(name) (CAUTION: to exit, hit ESC, then :q!)

Basic scripting in Linux

A basic loop on the command line

1 for ( ( i ; i <10; i ++ ) )2 do3 echo $ i4 done5 for i i n 1 3 7 996 do7 echo $ i8 done

Source: [1]

Capturing output

You can capture the output from a command

> seq 1 3123

Now let’s use that:

f o r i i n $ ( seq 1 3)do

echo $ idone

Basic scripting in Linux

Use for practical thingsRemember that ICF program sequence? How would we goabout starting 19 programs in sequence?

f o r program i n $ ( l s ∗ i c f . sas )do

sas $programdone

Advanced linux in 2 minutes

The gateway to everythingmanor try http://www.linuxmanpages.com or http://linux.die.net/man/

The toolkitI sedI grepI awkI regex (regular expressions)

Advanced scripting in Linux

Use for practical thingsWhat if I’m running 100s of programs, and trying to figure out ifany of them have errors?

f o r l o g f i l e s i n $ ( l s ∗ i c f . log )do

grep ERROR $ l o g f i l e sdone

Now let’s try it out

Next section

Workshop: High-performance computing for economists · Stampede (no. 6 on Top500 as of June 2013)...

Documents

Transcript of Workshop: High-performance computing for economists · Stampede (no. 6 on Top500 as of June 2013)...

Rectangular Wire Die Springs ISO-10243 Standard

INFO 7470 Overview of the Federal Statistical System John M. Abowd and Lars Vilhuber February 1, 2016.

Record Linking, II John M. Abowd and Lars Vilhuber April 2011 © 2011 John M. Abowd, Lars Vilhuber, all rights reserved.

Data access in North America Current state and future consequences William C. Block and Lars Vilhuber.

US Navy Course NAVEDTRA 10243 - Ear Training Manual for Musicians

JPRS ID: 10243 WORLDWIDE REPORT NARCOTICS AND … · title: jprs id: 10243 worldwide report narcotics and dangerous drugs subject: jprs id: 10243 worldwide report narcotics and dangerous

10243 - Lego · 2020. 6. 17. · 10224 10232 10243 LEGO and the LEGO logo are trademarks of the/sont des marques de commerce du /son marcas registradas de LEGO Group. ©2014, 2015

1231 10243 - congress.gov.ph

TPB Paper No. 10243 - Gov

INFO 4470/ILRLE 4470 Social and Economic Data Introduction John M. Abowd and Lars Vilhuber January 24, 2011.

SingleIncisionLaparoscopicSurgeryforAcuteAppendicitis ...downloads.hindawi.com/archive/2010/294958.pdf · laparoscopy. Up to 2 further 5mm DEXIDE (Covidien, Mansﬁeld, Massachusetts,

© 2007 John M. Abowd, Lars Vilhuber, all rights reserved An application of probabilistic matching Abowd and Vilhuber (2004), JBES.

Die Springs ISO 10243 · ISO 10243 Molle per stampi ... 2 Never select total deflections (preload + stroke) that are greater than the maximum deflection indicated in the catalogue.

sunum brosur2 eng - kalipyansan.com · o o 503 1.9 12,5 2.5 505 20 25 40 Heav,' Load Die Springs Red ISO 10243 No. 504 Extra - 12,5 6 3,2x Heavy Load Springs - Yelbw ISO 10243 Catab9

© John M. Abowd and Lars Vilhuber 2005, all rights reserved Record Linking, II John M. Abowd and Lars Vilhuber March 2005.

DIE SPRINGS to ISO 10243 - PROPLASTICA · 2018. 3. 5. · DIE SPRINGS to ISO 10243 SPRĘŻYNY DO TŁOCZNIKÓW zgodne z normą ISO 10243 1. S 2. Light duty springs - color green 3.

molle per stampi iso 10243 ISO 10243 die springswebshop.hagens.com/media/1079/die-springs-28062016.pdf · 2 _ total deflection (pre-load + working stroke) larger than the maximum

INFO 7470/ILRLE 7400 Statistical Tools: Edit and Imputation John M. Abowd and Lars Vilhuber March 25, 2013.

BS en 10243-1-1999 Steel Die for Forgings-Tolerances on Dimensions Part 1

US Navy music course - Ear Training Manual For Musicians NAVEDTRA 10243