LS-DYNA Analysis

8/17/2019 LS-DYNA Analysis

1/29

LS‐DYNA

Performance

Benchmarks

and Profiling

January 2009


2/29

2

Note

• The following research was performed under the HPC Advisory

Council activities

– AMD, Dell, Mellanox – HPC Advisory Council Cluster Center

• The participating members would like to thank LSTC for their

support and guidelines

• The participating members would like to thank Sharan Kalwani,

HPC Automotive specialist, for his support and guidelines• For more info please refer to

– www.mellanox.com, www.dell.com/hpc, www.amd.com

http://www.mellanox.com/http://www.dell.com/hpchttp://www.amd.com/http://www.amd.com/http://www.dell.com/hpchttp://www.mellanox.com/


3/29

3

LS-DYNA

• LS-DYNA

– A general purpose structural and fluid analysis simulation software

package capable of simulating complex real world problems

– Developed by the Livermore Software Technology Corporation (LSTC)

• LS-DYNA used by

– Automobile

– Aerospace

– Construction

– Military

– Manufacturing

– Bioengineering


4/29

4

LS-DYNA

• LS-DYNA SMP (Shared Memory Processing)

– Optimize the power of multiple CPUs within single machine

• LS-DYNA MPP (Massively Parallel Processing)

– The MPP version of LS-DYNA allows to run LS-DYNA solver over

High-performance computing cluster

– Uses message passing (MPI) to obtain parallelism

• Many companies are switching from SMP to MPP

– For cost-effective scaling and performance


5/29

5

Objectives

• The presented research was done to provide best practices

– LS-DYNA performance benchmarking

– Interconnect performance comparisons

– Ways to increase LS-DYNA productivity

– Understanding LS-DYNA communication pattern

– MPI libraries comparisons

– Power-aware consideration


6/296

Test Cluster Configuration

• Dell™ PowerEdge™ SC 1435 24-node cluster

• Quad-Core AMD Opteron™ Model 2358 processors (“ Barcelona” )

• Mellanox® InfiniBand ConnectX® DDR HCAs

• Mellanox® InfiniBand DDR Switch

• Memory: 16GB memory, DDR2 667MHz per node

• OS: RHEL5U2, OFED 1.3 InfiniBand SW stack

• MPI: HP MPI 2.2.7, Platform MPI 5.6.5

• Application: LS-DYNA MPP971

• Benchmark Workload

– Three Vehicle Collision Test simulation

– Neon-Refined Revised Crash Test simulation


7/297

Mellanox InfiniBand Solutions

• Industry Standard

– Hardware, software, cabling, management

– Design for clustering and storage interconnect

• Performance

– 40Gb/s node-to-node – 120Gb/s switch-to-switch

– 1us application latency

– Most aggressive roadmap in the industry

• Reliable with congestion management

• Efficient

– RDMA and Transport Offload

– Kernel bypass

– CPU focuses on application processing

• Scalable for Petascale computing & beyond

• End-to-end quality of service

• Virtualization acceleration

• I/O consolidation Including storage

Infin iBand Delivers the Lowest Latency

The InfiniBand Performance

Gap is Increasing

Fibre

Channel

Ethernet

60Gb/s

20Gb/s

120Gb/s

40Gb/s

240Gb/s

(12X)

80Gb/s

(4X)


8/298

8 November5, 2007

• Performance

– Quad-Core

• Enhanced CPU IPC

• 4x 512K L2 cache

• 2MB L3 Cache – Direct Connect Architecture

• HyperTransport™ technology

• Up to 24 GB/s

– Floating Point• 128-bit FPU per core

• 4 FLOPS/clk peak per core

– Memory

• 1GB Page Support

• DDR-2 667 MHz

• Scalability

– 48-bit Physical Addressing

• Compatibility

– Same power/thermal envelopes as Second-Generation AMD Opteron™ processor

PCI-E®Bridge

PCI-E®Bridge

I/O HubI/O Hub

USBUSB

PCIPCI

PCI-E®Bridge

PCI-E®Bridge

8 GB/S

8 GB/S

Dual ChannelReg DDR2

8 GB/S

8 GB/S

8 GB/S

Quad-Core AMD Opteron™ Processor


9/299

Dell PowerEdge Servers helping Simplify IT

• System Structure and Sizing Guidelines

– 24-node cluster build with Dell PowerEdge™ SC 1435 Servers

– Servers optimized for High Performance Computing environments

– Building Block Foundations for best price/performance and performance/watt

• Dell HPC Solutions

– Scalable Architectures for High Performance and Productivity

– Dell's comprehensive HPC services help manage the lifecycle requirements.

– Integrated, Tested and Validated Architectures

• Workload Modeling

– Optimized System Size, Configuration and Workloads

– Test-bed Benchmarks

– ISV Applications Characterization

– Best Practices & Usage Analysis


10/2910

LS-DYNA Performance Results - Interconnect

• InfiniBand high speed interconnect enables highest scalabil ity

– Performance gain with cluster size

• Performance over GigE is not scaling

– Slowdown occurs as number of processors increases beyond 16 nodes

LS-DYNA - 3 Vehic le Collis ion

01000

2000

300040005000

60007000

8000

4 ( 3 2

C o r

e s )

6 ( 4 8

C o r

e s )

8 ( 6 4

C o r

e s )

1 0 ( 8 0 C

o r e s )

1 2 ( 9 6 C

o r e s )

1 4 ( 1 1 2

C o r

e s )

1 6 ( 1 2 8

C o r

e s )

1 8 ( 1 4 4

C o r

e s )

2 0 ( 1 6 0

C o r

e s )

2 2 ( 1 7 6

C o r

e s )

2 4 ( 1 9 2

C o r

e s )

Number of Nodes

E

l a p s e d t i m e ( S e c o n d s )

InfiniBand GigE

LS-DYNA - Neon Refined Revised

0

100

200

300

400

500

600

700

4 ( 3 2

C o r

e s )

6 ( 4 8

C o r

e s )

8 ( 6 4

C o r

e s )

1 0 ( 8 0 C

o r e s )

1 2 ( 9 6 C

o r e s )

1 4 ( 1 1 2

C o r

e s )

1 6 ( 1 2 8

C o r

e s )

1 8 ( 1 4 4

C o r

e s )

2 0 ( 1 6 0

C o r

e s )

2 2 ( 1 7 6

C o r

e s )

2 4 ( 1 9 2

C o r

e s )

Num ber of Nodes

E

l a p s e d t i m e ( S e c o n d s )

In fin iBand GigE

Lower is better


11/2911


• InfiniBand outperforms GigE by up to 132%

– As node number increases, bigger advantage is expected

LS-DYNA - 3 Vehicle Collision

(InfiniBand vs GigE)

0%

20%

40%

60%

80%

100%120%

4 ( 3 2 C

o r e s )

6 ( 4 8

C o r

e s )

8 ( 6 4

C o r

e s )

1 0 ( 8 0 C

o r e s )

1 2 ( 9 6 C

o r e s )

1 4 ( 1 1 2

C o r

e s )

1 6 ( 1 2 8

C o r

e s )

1 8 ( 1 4 4

C o r

e s )

2 0 ( 1 6 0

C o r

e s )

2 2 ( 1 7 6

C o r

e s )

2 4 ( 1 9 2

C o r

e s )

Number of Nodes

P e r f

o r m a n c e A d v a n t a g e


(InfiniBand vs GigE)

0%

20%40%

60%

80%

100%

120%140%

4 ( 3 2

C o r

e s )

6 ( 4 8

C o r

e s )

8 ( 6 4

C o r

e s )

1 0 ( 8 0 C

o r e s )

1 2 ( 9 6 C

o r e s )

1 4 ( 1 1 2

C o r

e s )

1 6 ( 1 2 8

C o r

e s )

1 8 ( 1 4 4

C o r

e s )

2 0 ( 1 6 0

C o r

e s )

2 2 ( 1 7 6

C o r

e s )

2 4 ( 1 9 2

C o r

e s )

Number of Nodes

P e r f o

r m a n c e A d v a n t a g e


12/29

12

LS-DYNA Performance Results – CPU Affinity

• CPU affinity accelerates performance up to 10%

• Saves up to 177 seconds per simulation


(CPU Affinity vs Non-Affinity)

0

1000

2000

3000

40005000

6000

7000

4 ( 3 2

C o r

e s )

6 ( 4 8

C o r

e s )

8 ( 6 4

C o r

e s )

1 0 ( 8 0 C o

r e s )

1 2 ( 9 6 C o

r e s )

1 4 ( 1 1 2

C o r

e s )

1 6 ( 1 2 8

C o r

e s )

1 8 ( 1 4 4

C o r

e s )

2 0 ( 1 6 0

C o r

e s )

2 2 ( 1 7 6

C o r

e s )

2 4 ( 1 9 2

C o r

e s )

Number of Nodes

E

l a p s e d t i m e ( S

e c o n d s )

CPU Affinity Without Affinity

Lower is better


13/29

13

LS-DYNA Performance Results - Productivity

• InfiniBand increases productivity by allowing multiple jobs to run simultaneously – Providing required productivity for virtual vehicle design

• Three cases are presented

– Single job over the entire systems (with CPU affinity)

– Two jobs, each on a single CPU per server (job placement , CPU affinity)

– Four jobs, each on two CPU cores per CPU per server (job placement , CPU affinity)

• Four jobs per day increases productivity by 97% for Neon Refined Revised, 57% for 3 Car coll ision case

• Increased number of parallel processes (jobs) increases the load on the interconnect

– High speed and low latency interconnect solution is required for gaining high productivity


0100

200300400500600700800900

1000

4 ( 3 2 C

o r e s )

8 ( 6 4 C o

r e s )

1 2 ( 9 6 C o

r e s )

1 6 ( 1 2 8

C o r

e s )

2 0 ( 1 6 0

C o r

e s )

2 4 ( 1 9 2

C o r

e s )

Number of Nodes

J o b s p e r D a y

1 Job 2 Paral lel Jobs 4 Paral lel Jobs


010

2030405060708090

4 ( 3 2 C

o r e s )

8 ( 6 4 C

o r e s )

1 2 ( 9 6 C

o r e s )

1 6 ( 1 2 8

C o r

e s )

2 0 ( 1 6 0

C o r

e s )

2 4 ( 1 9 2

C o r

e s )

Number of Nodes

J o b s p e r D a y

1 Job 2 Paral lel Jobs 4 Paral lel Jobs

Higher is better


14/29

14

LS-DYNA MPI Profiliing

10000000

100000000

1E+09

1E+10

1E+11

1E+12

[ 0 .

. 6 4 B

]

[ 6 4 . . 2 5 6

B ]

[ 2 5 6 B .

. 1 K B

]

[ 1 .

. 4 K B

]

[ 4 . . 1 6 K

B ]

[ 1 6 . . 6 4 K

B ]

[ 6 4 . . 2 5

6 K B ]

[ 2 5 6 K

B . . 1 M ]

[ 1 . . 4 M ]

[ 4 M . . i n f i n i t y

]

Message Size

T o t a l S i z e ( M B )

4nodes 8nodes 12nodes 16nodes 20nodes 24nodes

LS-DYNA Profiling – Data Transferred

(3 Vehicle Collision)

• Majority of data transfer is done via 256B-4KB message size


15/29

15

LS-DYNA Profiling – Message Distribution

LS-DYNA MPI Profil iing

1

10

100

1000

10000

100000

1000000

10000000

100000000

1000000000

[ 0 . . 6 4 B

]

[ 6 4 . .

2 5 6 B ]

[ 2

5 6 B .

. 1 K B

]

[ 1 . . 4 K B

]

[ 4 . . 1 6 K

B ]

[ 1 6 . .

6 4 K B

]

[ 6 4

. . 2 5 6

K B ]

[ 2

5 6 K B

. . 1 M ]

[ 1 . . 4 M ]

[ 4 M .

. i n f i n

i t y ]

Message Size

N u

m b e r o f M e s s a

g e s



• Majority of the messages are in the range of 2B-4KB – 2B-256B for synchronization, 256B-4KB for data communications


16/29

16

LS-DYNA MPI Prof ili ing

0%

10%

20%

30%

40%

50%

60%

70%

[0..64] [65..256] [257..1024] [1025..4096] [4097..16384]

Message Size

% o f t o t a l m e s s a g e

s


LS-DYNA Profiling – Message Distribution


• As number of nodes scales, percentage of small messages increases

• percentage of 256-1KB messages is relatively consistent with cluster size – Actual number increases with cluster size,


17/29

17

LS-DYNA Profiling – MPI Collectives

• Two key MPI collective functions in LS-DYNA

– MPI_AllReduce

– MPI_Bcast

• Account for the majority of MPI communication overhead

MPI Collectives

0%

10%

20%

30%

40%

50%60%

70%

4 ( 3 2

C o r e s

)

8 ( 6 4

C o r e s

)

1 2 ( 9 6 C

o r e s )

1 6 ( 1 2 8

C o r e s

)

2 0 ( 1 6 0

C o r e s

)

2 4 ( 1 9 2

C o r e s

)

Number of Nodes

% o f

T o t a l O v e r h e a d

( m s )

MPI_AllReduce MPI_Bcast


18/29

18

MPI Collective Benchmarking

MPI_Bcast

0

5

10

15

20

25

30

0 1 2 4 8 16 32 64 128 256 512

Message Size

L a t

e n c y ( u s e c )

HP-MPI Platform MPI

MPI_AllReduce

0

20

40

60

80

100

120

0 4 8 16 32 64 128 256 512

Mes sage Size

L a t e n c y ( u s e c )

HP-MPI Platform MPI

• MPI collective performance comparison

– Two frequently called collection operations in LS-DYNA were benchmarked

• MPI_Allreduce

• MPI_Bcast

– Platform MPI shows better latency for AllReduce operation


19/29

19

LS-DYNA with Different MPI Libraries

• LS-DYNA performance Comparison – Each MPI library shows different benefits for latency and collectives

– As such, HP-MPI and Platform MPI shows comparable performance

LS-DYNA - 3 Vehicle Col lis ion

1000

2000

3000

40005000

6000

7000

4 ( 3 2

C o r e s )

6 ( 4 8

C o r e s )

8 ( 6 4

C o r e s )

1 0 ( 8 0 C o

r e s )

1 2 ( 9 6 C o

r e s )

1 4 ( 1 1 2

C o r e

s )

1 6 ( 1 2 8

C o r e

s )

1 8 ( 1 4 4

C o r e

s )

2 0 ( 1 6 0

C o r e

s )

2 2 ( 1 7 6

C o r e

s )

2 4 ( 1 9 2

C o r e

s )

Num ber of Nodes

E l a p s e d t i m e ( S

e c o n d s )

Platform MPI HP-MPI

LS-DYNA - Neon Ref ined Rev ised

100150200

250300350400

450500550

4 ( 3 2 C o r

e s )

6 ( 4 8

C o r e s )

8 ( 6 4

C o r e s )

1 0 ( 8 0 C o r

e s )

1 2 ( 9 6 C o r

e s )

1 4 ( 1 1 2

C o r e s )

1 6 ( 1 2 8

C o r e s )

1 8 ( 1 4 4

C o r e s )

2 0 ( 1 6 0

C o r e s )

2 2 ( 1 7 6

C o r e s )

2 4 ( 1 9 2

C o r e s )

Num ber of Nodes


e c o n d s )

Plat form MPI HP-MPI

Lower is better


20/29

20

LS-DYNA Profiling Summary - Interconnect

• LS-DYNA was profiled to determine networking dependency

• Majority of data transferred between compute nodes

– Done with 256B-4KB message size, data transferred increases with cluster size

• Most used message sizes –


21/29

21

Test Cluster Configuration – System Upgrade

• The following results were achieved after system upgrade (changes are in green)

– Dell PowerEdge SC 1435 24-node cluster

– Quad-Core AMD Opteron™ Model 2382 processors (“Shanghai”) (vs “Barcelona” in previous

configuration)

– Mellanox® InfiniBand ConnectX® DDR HCAs

– Mellanox® InfiniBand DDR Switch

– Memory: 16GB memory, DDR2 800MHz per node (vs 667MHz in previous configuration)

– OS: RHEL5U2, OFED 1.3 InfiniBand SW stack

– MPI: HP MPI 2.2.7, Platform MPI 5.6.5

– Application: LS-DYNA MPP971

– Benchmark Workload

• Three-Car Crash Test simulation

• Neon-Refined Revised Crash Test simulation


22/29

22

• Performance

– Quad-Core

• Enhanced CPU IPC

• 4x 512K L2 cache

• 6MB L3 Cache – Direct Connect Architecture

• HyperTransport™ technology

• Up to 24 GB/s peak per processor

– Floating Point• 128-bit FPU per core

• 4 FLOPS/clk peak per core

– Integrated Memory Controller

• Up to 12.8 GB/s

• DDR2-800 MHz or DDR2-667 MHz

• Scalability

– 48-bit Physical Addressing

• Compatibility

– Same power/thermal envelopes as 2nd / 3rd generation AMD Opteron™ processor

22 November5, 2007

PCI-E®Bridge

PCI-E®Bridge

I/O HubI/O Hub

USBUSB

PCIPCI

PCI-E®Bridge

PCI-E®Bridge

8 GB/S

8 GB/S

Dual ChannelReg DDR2

8 GB/S

8 GB/S

8 GB/S

Quad-Core AMD Opteron™ Processor


23/29

23

Performance Improvement

• Upgraded AMD CPU and DDR-2 Memory

• LS-DYNA run time decreased by more than 20%

– Leveraging InfiniBand 20Gb/s for higher scalability


0

1000

2000

3000

4000

5000

6000

7000

4 ( 3 2

C o r e s )

6 ( 4 8

C o r e s )

8 ( 6 4

C o r e s )

1 0 ( 8 0 C o

r e s )

1 2 ( 9 6 C o

r e s )

1 4 ( 1 1 2

C o r e

s )

1 6 ( 1 2 8

C o r e

s )

1 8 ( 1 4 4

C o r e

s )

2 0 ( 1 6 0

C o r e

s )

2 2 ( 1 7 6

C o r e

s )

2 4 ( 1 9 2

C o r e

s )

Number of Nodes

E l a p s e d t i m e ( S e c o n d s )

Barcel ona Shanghai


0

100

200

300

400

500

600

4 ( 3 2

C o r e s )

6 ( 4 8

C o r e s )

8 ( 6 4

C o r e s )

1 0 ( 8 0 C o

r e s )

1 2 ( 9 6 C o

r e s )

1 4 ( 1 1 2 C o

r e s )

1 6 ( 1 2 8 C o

r e s )

1 8 ( 1 4 4 C o

r e s )

2 0 ( 1 6 0 C o

r e s )

2 2 ( 1 7 6 C o

r e s )

2 4 ( 1 9 2 C o

r e s )

Number of Nodes


e c o n d s )

Barcel ona Shanghai

Lower is better


24/29

24

Maximize LS-DYNA Productivity

• Scalable latency of InfiniBand and latest Shanghaiprocessor deliver scalable LS-DYNA performance


0

20

40

60

80

100

120

4 ( 3 2

C o r e s

)

8 ( 6 4

C o r e s

)

1 2 ( 9 6 C

o r e s )

1 6 ( 1 2 8

C o r e s

)

2 0 ( 1 6 0

C o r e s

)

2 4 ( 1 9 2

C o r e s

)

Number of Nodes

J o

b s p e r D a y

1 Job 2 Parallel Jobs 4 Paralle l Jobs 8 Parallel Jobs


0

200

400

600

800

1000

1200

1400

4 ( 3 2

C o r e s

)

8 ( 6 4

C o r e s

)

1 2 ( 9 6 C

o r e s )

1 6 ( 1 2 8

C o r e s

)

2 0 ( 1 6 0

C o r e s

)

2 4 ( 1 9 2

C o r e s

)

Number of Nodes

J o

b s p e r D a y

1 Job 2 Parallel Jobs 4 Parallel Jobs 8 Parallel Jobs

Higher is better


25/29

25

LS-DYNA with Shanghai Processors

• “ Shanghai” processors provides higher performancecompared to “ Barcelona’

LS-DYNA - 3 Vehicle Collision(Shanghai vs Barcelona)

0%

5%

10%

15%

20%

25%

30%

4 ( 3 2

C o r e s )

8 ( 6 4

C o r e s )

1 2 ( 9 6 C

o r e s )

1 6 ( 1 2 8

C o r e s )

2 0 ( 1 6 0

C o r e s )

2 4 ( 1 9 2

C o r e s )

Number of Nodes

%

o f m o r e j o b s p e r d a y

1 Job 2 Parallel Jobs 4 Par allel Jobs


26/29

26


• InfiniBand 20Gb/s vs 10GigE vs GigE

• InfiniBand 20Gb/s (DDR) outperforms 10GigE and GigE in all test cases

– Reducing run time by up to 60% versus 10GigE and 61% vs GigE

• Performance loss shown beyond 16 nodes with 10GigE and GigE

• InfiniBand 20Gb/s maintain scalability with cluster size


(HP-MPI)

0

100

200

300

400

500

600

4 ( 3 2

C o r e s

)

8 ( 6 4

C o r e s

)

1 2 ( 9 6 C

o r e s )

1 6 ( 1 2 8

C o r e s

)

2 0 ( 1 6 0

C o r e s

)

2 4 ( 1 9 2

C o r e s

)

Number of Nodes

E l a p s

e d t i m e ( S e c o n d s )

GigE 10Gig E Inf iniBan d


(HP-MPI)

0

1000

2000

3000

4000

5000

6000

4 ( 3 2

C o r e s

)

8 ( 6 4

C o r e s

)

1 2 ( 9 6 C

o r e s )

1 6 ( 1 2 8

C o r e s

)

2 0 ( 1 6 0

C o r e s

)

2 4 ( 1 9 2

C o r e s

)

Number of Nodes

E l a p s

e d t i m e ( S e c o n d s )

GigE 10GigE In finiBand

Lower is better


27/29

27

Power Consumption(InfiniBand vs 10GigE vs GigE)

0

500

1000

1500

2000

2500

3000

3500

4000

4500

3 Vehicle Collision Neon Refined Revised

W

h p e r J o b

GigE 10GigE InfiniBand

Power Consumption Comparison

• InfiniBand also enables power efficient simulations

– Reducing power/job by up to 62%!

62%

50%

24-node comparison


28/29

28

Conclusions

• LS-DYNA is widely used to simulate many real-world problems

– Automotive crash-testing and finite-element simulations

– Developed by Livermore Software Technology Corporation (LSTC)

• LS-DYNA performance and productivity relies on – Scalable HPC systems and interconnect solutions

– Low latency and high throughput interconnect technology

– NUMA aware application for fast access to local memory

– Reasonable job distribution can dramatically improve productivity

• Increasing number of jobs per day while maintaining fast run time

• Interconnect comparison shows

– InfiniBand delivers superior performance and productivity in every cluster size – Scalability requires low latency and “zero” scalable latency

– Lowest power consumption was achieved with InfiniBand

• Saving in system power, cooling and real-estate


29/29

29

Thank YouHPC Advisory [email protected]

All trademarks are proper ty o f their respecti ve owners . All information i s provided “ As-Is” wi thout any kind o f warranty. The HPC Advisory Counc il makes no representat ion to the accuracy and

completeness of the information contained herein. HPC Advisory Council Mellanox undertakes no duty and assumes no obligation to update or correct any information presented herein

LS-DYNA Analysis

Documents

Transcript of LS-DYNA Analysis