1

CS140: Parallel Scientific

Computing

Class Introduction

Tao Yang, UCSB

Tuesday/Thursday. 11:00-12:15 GIRV 1115

2

CS 140 Course Information

• Instructor: Tao Yang (tyang@cs).

Office Hours: T/Th 10-11(or email me for appointments or

just stop by my office). HFH building, Room 5113

• Supercomputing consultant: Kadir Diri and Stefan Boeriu

• TA: Xin Jin [xin_jin@cs]. Steven Bluen [sbluen153@yahoo]

• Text book

An Introduction to Parallel Programming" by Peter

Pacheco, 2011, Morgan Kaufmann Publisher

• Class slides/online references:

http://www.cs.ucsb.edu/~tyang/class/140s14

• Discussion group: registered students are invited to join a

google group

3

Introduction

• Why all computers must be parallel computing

• Why parallel processing?

Large Computational Science and

Engineering (CSE) problems require

powerful computers

Commercial data-oriented computing also

needs.

• Why writing (fast) parallel programs is hard

• Class Information

4

All computers use parallel computing

• Web+cloud computing

Big corporate computing

• Enterprise computing

• Home computing

Desktops, laptops,

handhelds & phones

Drivers behind high performance computing

1

10

100

1,000

10,000

100,000

1,000,000

Jun-9

3

Jun-9

4

Jun-9

5

Jun-9

6

Jun-9

7

Jun-9

8

Jun-9

9

Jun-0

0

Jun-0

1

Jun-0

2

Jun-0

3

Jun-0

4

Jun-0

5

Jun-0

6

Jun-0

7

Jun-0

8

Jun-0

9

Jun-1

0

Jun-1

1

Jun-1

2

Jun-1

3

Jun-1

4

Jun-1

5

# p

rocesso

rs

.

Parallelism

Big Data Drives Computing Need Too

Zettabyte = 270 ~ 1 billion Terabytes

Exabyte = 1 million Terabytes

Examples of Big Data

• Web search/ads (Google, Bing, Yahoo, Ask)

10B+ pages crawled -> indexing 500-1000TB /day

10B+ queries+pageviews /day 100+ TB log

• Social media

Facebook: 3B content items shared. 3B- “like”.

300M photo upload. 500TB data ingested/day

Youtube: A few billion views/day. Millions of TB.

• NASA

12 data centers, 25,000 datasets. Climate weather

data: 32PB 350PB

NASA missions stream 24TB/day. Future space

data demand: 700 TB/second

8

Metrics in Scientific Computing World

• High Performance Computing (HPC) units are:

Flop: floating point operation, usually double precision

unless noted

Flop/s: floating point operations per second

Bytes: size of data (a double precision floating point

number is 8)

• Typical sizes are millions, billions, trillions…

• Current fastest (public) machines in the world

Up-to-date list at www.top500.org

Top one has 33.86 Pflop/s using 3.12 millions of cores

9

Typical sizes are millions, billions, trillions…

Mega Mflop/s = 106 flop/sec Mbyte = 220 ~ 106 bytes

Giga Gflop/s = 109 flop/sec Gbyte = 230 ~ 109 bytes

Tera Tflop/s = 1012 flop/sec Tbyte = 240 ~ 1012 bytes

Peta Pflop/s = 1015 flop/sec Pbyte = 250 ~ 1015 bytes

Exa Eflop/s = 1018 flop/sec Ebyte = 260 ~ 1018 bytes

Zetta Zflop/s = 1021 flop/sec Zbyte = 270 ~ 1021 bytes

Yotta Yflop/s = 1024 flop/sec Ybyte = 280 ~ 1024 bytes

Rank Site System Cores

Rmax

(TFlop/s)

Rpeak

(TFlop/s)

Power

(kW)

1

NSCC

China

MilkyWay

-2 - Intel

Xeon E5

2.2GHz

NUDT

3120000 33862.7 54902.4 17808

2 DOE/SC/Oak

Ridge National

Laboratory

United States

Titan

AMD

Opteron,

2.2GHz

NVIDIA

K20x

Cray Inc.

560640 17590.0 27112.5 8209

3 DOE/NNSA/L

LNL

United States

Sequoia -

BlueGene/

Q, Power

BQC 16C

1.60 GHz,

Custom

IBM

1572864 16324.8 20132.7 7890

From www.top500.org (Nov 2013)

11

Why parallel computing? Can a single high

speed core be used?

• Chip density is continuing increase ~2x every 2 years

• Clock speed is not

• Number of processor cores may double instead

• Power is under control, no longer growing

0

1

10

100

1000

10000

100000

1000000

10000000

1970 1975 1980 1985 1990 1995 2000 2005 2010

Transistors (Thousands) Frequency (MHz) Power (W) Cores

Can we just use one machine with many cores and

big memory/storage?

Technology trends against increasing memory per core

• Memory performance is not keeping pace, even

• Memory density is doubling every three years

• Storage costs (dollars/Mbyte) are dropping gradually

• have to use a distributed architecture for many highend

computing

13

Impact of Parallelism

• All major processor vendors are producing multicore

chips

Every machine is a parallel machine

To keep doubling performance, parallelism must double

• Which commercial applications can use this

parallelism?

Do they have to be rewritten from scratch?

• Will all programmers have to be parallel programmers?

New software model needed

Try to hide complexity from most programmers –

eventually

• Computer industry betting on this big change, but

does not have all the answers

Slide source: Demmel/Yelick

14

Roadmap

• Why all computers must be parallel computing

• Why parallel processing?

Large Computational Science and

Engineering (CSE) problems require

powerful computers

Commercial data-oriented computing also

needs.

• Why writing (fast) parallel programs is hard

• Class Information

15

Examples of Challenging Computations That

Need High Performance Computing

• Science Global climate modeling

Biology: genomics; protein folding; drug design

Astrophysical modeling

Computational Chemistry

Computational Material Sciences and Nanosciences

• Engineering Semiconductor design

Earthquake and structural modeling

Computation fluid dynamics (airplane design)

Combustion (engine design)

Crash simulation

• Business Financial and economic modeling

Transaction processing, web services and search engines

• Defense Nuclear weapons -- test by simulations

Cryptography Slide source: Demmel/Yelick

16

Economic Impact of High Performance Computing

• Airlines:

System-wide logistics optimization on parallel systems.

Savings: approx. $100 million per airline per year.

• Automotive design:

Major automotive companies use 500+ CPUs for:

– CAD-CAM, crash testing, structural integrity and aerodynamics.

– One company has 500+ CPU parallel system.

Savings: approx. $1 billion per company per year.

• Semiconductor industry:

Semiconductor firms use large systems (500+ CPUs)

for

– device electronics simulation and logic validation

Savings: approx. $1 billion per company per year. Slide source: Demmel/Yelick

17

Global Climate Modeling

• Problem is to compute:

f(latitude, longitude, elevation, time)

“weather” = (temperature, pressure, humidity,

wind velocity)

• Approach:

Discretize the domain, e.g., a measurement point every 10 km

Devise an algorithm to predict weather at time step

• Uses:

- Predict major events, e.g., hurricane, El Nino

- Use in setting air emissions standards

- Evaluate global warming scenarios

Slide source: Demmel/Yelick

18

Global Climate Modeling: Computational

Requirements

• One piece is modeling the fluid flow in the atmosphere

Solve numerical equations

– Roughly 100 Flops per grid point with 1 minute timestep

• Computational requirements:

To match real-time, need 5 x 1011 flops in 60 seconds = 8

Gflop/s

Weather prediction (7 days in 24 hours) 56 Gflop/s

Climate prediction (50 years in 30 days) 4.8 Tflop/s

To use in policy negotiations (50 years in 12 hours) 288

Tflop/s

• To double the grid resolution, computation is 8x to 16x

Slide source: Demmel/Yelick

Mining and Search for Big Data

• Identify and discover information from a massive amount of data

• Business intelligence required

by many companies/organizations

3/30/2014 20

Multi-tier Web Services: Search Engine

Network

Cache

Frontend

Client queries Traffic load balancer

Cache Cache

Cache

Frontend Frontend Frontend

Index match

Tier 1

Index match

Tier 2

Document

Abstract Document

Abstract Document

Abstract Document

description

Ranking Ranking Ranking Ranking Ranking Rank

Server

Search

Suggestion

Advertisement

Engine cluster

21

IDC HPC Market Study

• International Data Corporation (IDC) is an American

market research, analysis and advisory firm

• HPC covers all servers that are used for highly

computational or data intensive tasks

HPC revenue for 2014 exceeded $12B

forecasting ~7% growth over the next 5 years

Source: IDC July 2013

Supercomputer

segment: IDC defines as

systems $500,000 and up.

Motif/Dwarf: Common Computational Methods (Red Hot Blue Cool)

Em

be

d

SP

EC

DB

Ga

me

s

ML

HP

C

Health Image Speech Music Browser

1 Finite State Mach.

2 Combinational

3 Graph Traversal

4 Structured Grid

5 Dense Matrix

6 Sparse Matrix

7 Spectral (FFT)

8 Dynamic Prog

9 N-Body

10 MapReduce

11 Backtrack/ B&B

12 Graphical Models

13 Unstructured Grid

What do compute-intensive applications have in common?

Types of Big Data Representation

• Text, multi-media,

social/graph data

• Represented by

weighted feature vectors,

matrices, graphs

The Web

Social graph

24

Basic Scientific Computing Algortihms

• Matrix-vector multiplication.

• Matrix-matrix multiplication.

• Direct method for solving a linear equation.

Gaussian Elimination.

• Iterative method for solving a linear equation.

Jacobi, Gauss-Seidel.

• Sparse linear systems and differential equations.

25

Roadmap

• Why all computers must be parallel computing

• Why parallel processing?

Large Computational Science and

Engineering (CSE) problems require

powerful computers

Commercial data-oriented computing also

needs.

• Why writing (fast) parallel programs is hard

• Class Information

26

Principles of Parallel Computing

• Finding enough parallelism (Amdahl’s Law)

• Granularity

• Locality

• Load balance

• Coordination and synchronization

• Performance modeling

All of these things makes parallel programming

even harder than sequential programming.

27

Overhead of Parallelism

• Given enough parallel work, this is the biggest

barrier to getting desired speedup

• Parallelism overheads include:

cost of starting a thread or process

cost of accessing data, communicating shared data

cost of synchronizing

extra (redundant) computation

• Each of these can be in the range of milliseconds

(=millions of flops) on some systems

• Tradeoff: Algorithm needs sufficiently large units of

work to run fast in parallel (i.e. large granularity), but

not so large that there is not enough parallel work

Slide source: Demmel/Yelick

28

Locality and Parallelism

• Large memories are slow, fast memories are small

• Slow accesses to “remote” data or communicate with other machines

• Algorithm should do most work on local data, and minimize communication

overhead

Proc Cache

L2 Cache

L3 Cache

Memory

Conventional

Storage

Hierarchy

Proc Cache

L2 Cache

L3 Cache

Memory

Proc Cache

L2 Cache

L3 Cache

Memory

poten

tial

interco

nn

ects

Slide source: Demmel/Yelick

29

Load Imbalance

• Load imbalance is the time that some processors

in the system are idle due to

insufficient parallelism (during that phase)

unequal size tasks

• Examples: tree-structured computations.

Unstructured problems

• Algorithm needs to balance load

Sometimes can determine work load, divide up

evenly, before starting

– “Static Load Balancing”

Sometimes work load changes dynamically, need to

rebalance dynamically

– “Dynamic Load Balancing”

Slide source: Demmel/Yelick

30

Improving Real Performance

0.1

1

10

100

1,000

2000 2004 T

erafl

op

s 1996

Peak Performance grows exponentially

But efficiency (the performance relative to the

hardware peak) has declined

was 40-50% on the vector supercomputers of

1990s

now as little as 5-10% on parallel supercomputers

of today

Close the gap through ...

Computing methods and algorithms that achieve

high performance on a single processor and scale

to thousands of processors

More efficient programming models and tools for

massively parallel supercomputers

Performance

Gap

Peak Performance

Real Performance

Slide source: Demmel/Yelick

31

Roadmap

• Why all computers must be parallel computing

• Why parallel processing?

Large Computational Science and

Engineering (CSE) problems require

powerful computers

Commercial data-oriented computing also

needs.

• Why writing (fast) parallel programs is hard

• Class Information

32

Course Objective

In depth understanding of:

• When is parallel computing useful?

• Understanding of parallel computing hardware

options.

• Overview of programming models (software) and

tools and performance analysis

• Some important parallel applications and the

algorithms for scientific/data-intensive computing

33

Course Topics

• High performance computing

Basics of computer architecture, clusters&cloud

systems. Storage.

• Parallel programming models, software/libraries

Task graph computation. Embarrassingly parallel,

divide-and-conquer, and pipelining.

Partitioning and mapping of program/data for

shared memory vs distributed memory

Threads, MPI, MapReduce/Hadoop, and openMP if

time permits

• Patterns of parallelism. Optimization techniques for

parallelization and performance

• Core computing algorithms in scientific and data-

intensive web applications

34

Class Computing Resource

TSCC Cluster at San Diego Supercomputer Center

Computing: Up to 512 cores.

Node architecture

• 16 cores/machine, 2.6GHz Intel Xeon E5-2670 (Sandy Bridge)

• Memory: 64GB per machine

Network: 10GbE (QDR InfiniBand optional)

Storage: 100GB/user with a backup.

200TB shared scratch space

available to all users.

35

Class Computing Resource

• Triton Shared Computing Cluster (TSCC)

accounts:

Apply in week 1

Get a class account in Triton by emailing your

name, UCSB email, and ssh public key with subject

"CS140 ssh key" to scc@oit.ucsb.edu .

Instructions on generating ssh keys can be found in

class webpage

CSIL TSCC

Cluster at

San Diego

Your

laptop

36

Prerequisites and Misc Info

• Prerequisites

Data structure and algorithms (CS 130A).

– Graph, tree, stack, queue data structures

– Sorting. Shortest path algorithms. Algorithm complexity

Programming experience with C and Java on Linux.

– OS and programming experience!

Linear algebra (e.g. Math 5A or 4A)

– Vectors, matrix. Linear equation solving.

Basic computer architecture (CPUs, cache, memory)

• Class material is updated in

http://www.cs.ucsb.edu/~tyang/class/140s14

• Text book source code:

http://www.cs.usfca.edu/~peter/ipp/

• CS140 class discussion group at Google

37

Course Workload and Challenges

• Workload and weighting

2-person group homework (55%). Exams (45%).

4-5 homework and programming assignments. One group

interview.

Midterm (May 6) Final (June 11?)

• Challenges

Textbook/documents may not represent the latest

development:

– Parallel system is complex. Big data/large scale computing is hard

– Parallel computing technology evolves fast in last ten years.

– Documentation is weak (e.g. Hadoop Mapreduce)

Reading with self-searching of web material is needed.