Purdue University - Linux Clusters Institute · Purdue University The 6th International Conference...
Transcript of Purdue University - Linux Clusters Institute · Purdue University The 6th International Conference...
Parallel MultiParallel Multi--Zone Methods for LargeZone Methods for Large--Scale Multidisciplinary Computational Scale Multidisciplinary Computational
Physics SimulationsPhysics Simulations
Ding Li, Ding Li, Guoping XiaGuoping Xia and Charles L. Merkleand Charles L. Merkle
Purdue UniversityPurdue University
The 6th International Conference on Linux ClustersThe 6th International Conference on Linux ClustersThe HPC Revolution 2005The HPC Revolution 2005
Chapel Hill, NC, April 25Chapel Hill, NC, April 25--28, 200428, 2004
Presentation OutlinePresentation Outline
MultidisciplinaryMultidisciplinaryNumerical Analysis SystemNumerical Analysis SystemGEMS codeGEMS codeGeneralized Equations of MotionGeneralized Equations of MotionLinux Cluster and Benchmarks Linux Cluster and Benchmarks Parallel ImplementationParallel ImplementationRepresentative ApplicationsRepresentative Applications
Multidisciplinary Computational Multidisciplinary Computational PhysicsPhysics
•Multi-physicsstructures, plasma dynamics, fluid dynamics,
electromagnetics, radiative energy transfer and neutron transport
•Different approaches
loosely coupled and individual codes
closely coupled and solved simultaneously
•Unified Framework General conservation law
Data Mining/Data Mining/VisualizationVisualization
CAD
GRID GENERATOR
PGRIDPGRID
Data Repository
G E
M S
G E
M S
G E
M S
PropertyPropertyModulesModules
Numerical Analysis SystemNumerical Analysis System
Purdue University - School of Mechanical Engineering
Fluid-Solid Model
GEMS codeGEMS code
∑∑ ΩΩ =−+∂
∂Γ
N
ii
Visivn
N
ii
Invin
p VSAFAFVt
Q,,
General EquationPreconditioned,Multiple-TimeAlgorithms
Preconditioned,Multiple-TimeAlgorithms
Structured-Unstructured Grids
Cluster ComputingCluster Computing
Multiple Physical Zones
G E
M S
G E
M S
G E
M S
στρρ rrrrrrr
•∇+•∇+−∇=•∇+∂∂ pVV
tV
−= )(
312 etraceSM
rrrrrrrrδµσ
ElectromagneticsElectromagnetics
Generalized EquationsGeneralized Equations
0=Φ∇+×∇+•∇+∂∂
+∂
∂Γ CD
p FFtQQ
τ
Generic set of partial differential equations:Generic set of partial differential equations:
0=∫ ΩΦ∇+∫ Ω×∇+∫ Ω•∇+∫ Ω∂∂
+Ω∂
∂Γ
ΩΩΩΩddFdFd
tQQ
CDp
τ
0=∫ ΣΦ+∫ Σ×+∫ Σ•+
∫ Ω
∂∂
+Ω∂
∂Γ
Ω∂Ω∂Ω∂ΩdndFndFnQd
tQ
CDp
τ
Normal Flux Tangential Flux Scale
Number of Partial Differential Number of Partial Differential Equations in Various FieldsEquations in Various Fields
MultiMulti--Physics Physics Zone MethodZone Method
Cluster2Cluster2Cluster2Cluster3Cluster3Cluster3
Cluster1Cluster1Cluster1
• Distinct Physics Zones•Different media•Different equations
• Parallel Processing
•Each zone divided into sub-clusters
• Load Balancing•Number of equations•Size of grids
Linux ClustersLinux ClustersSimbaSimba (2001)(2001)51 nodes:Single P4 1.8 51 nodes:Single P4 1.8 GhzGhz CPU:1Gb CPU:1Gb RAMsRAMs:10/100 :10/100 EthernetEthernetRedhat9.0, Redhat9.0, Lahey Lahey Fortran Compiler, MPICH1.2.4, PBSFortran Compiler, MPICH1.2.4, PBSMacbethMacbeth (2005)(2005)98 nodes: dual AMD 98 nodes: dual AMD OpteronOpteron 1.8 1.8 GhzGhz CPU: 4GB CPU: 4GB RAMsRAMs::InfinibandInfiniband interconnect:4X interconnect:4X InfinibandInfiniband network network fabric (10Gbps)fabric (10Gbps)Redhat Redhat Enterprise, Intel, PGI and Enterprise, Intel, PGI and PathscalePathscale Fortran Fortran Compiler, MPICH1.2.6,Compiler, MPICH1.2.6,PBSProPBSPro
Simba Simba vs. vs. MacbethMacbeth2D turbulent flow w/ 0.5 Million grid cells
Number of Processors
WTi
me/
cells
/iter
atio
ns
Wal
lTim
e(s
)
0 10 20 30 40
0.0E+00
1.0E-04
2.0E-04
3.0E-04
4.0E-04
0
500
1000
1500
2000
2500
wall time (Simba,Lahey)
wtime/cells/iterations (Macbeth.Intel)
wall time (Macbeth,Intel)
wtime/cells/iterations (Simba,Lahey)
Number of Processors
WTi
me/
cells
/iter
atio
ns(s
ec.)
0 10 20 30 40 50 60
6.0E-05
8.0E-05
1.0E-04
1.2E-04
1.4E-04wtime/cells/iterations/Processors
Pathscale
PGI
Intel
Intel, Intel, Pathscale Pathscale vs. PGIvs. PGI
3D flow with 3D flow with 3.375Million grid cells3.375Million grid cells
Parallel Computing & PartitioningParallel Computing & Partitioning
N01
N02
N03
N04…N10
10 Partitions
Parallel Data StructureParallel Data Structure
Definitions:Definitions:Interface:The face adjoined by two different partitionsSending Data:The cells of current partition adjoined to the interfaceReceiving Data:The cells of all partitions except current partition adjoined to the interface
Current partition
Receiving data
Sending data
Interface
Exchanging matrixExchanging matrix•Zero diagonal: no data exchange inside a partition
•Rows represent the number of sending data
•Columns represent the number of receiving data
•Sum of rows are total number of data received in the partition of the row
•Sum of columns are total number of data sent in the partition of the column
•The number of each element is the number of data sent by the column partition to the row partition
0280030000800300030500050
1 2 3 4 5
1
2
3
4
5
Sending partition (processor)Receiving partition (processor)
Cell list for sending and receivingCell list for sending and receiving
535251232221201818103
Cell list for sending in partition 3
Cell list of receiving in partition 5
Cell list of receiving in partition 2
87654321
10987654321
Partition 1
Partition 4
Representative ApplicationsRepresentative ApplicationsConstant Volume Combustion Turbine SystemConstant Volume Combustion Turbine System
Calculation in only one sector is necessary, since flow in other sectors experience same condition, at different time
Boundary condition at sector interface is provided by the solution in the same sector at earlier time step, which is determined by the firing order
Pulse Detonation Engine and Pulse Detonation Engine and Turbine Interaction ResearchTurbine Interaction Research
Tem
pera
ture
&
Tem
pera
ture
&
Pre
ssu
re C
on
tou
rsP
ress
ure
Co
nto
urs
SummarySummary
Unified parallel framework for dealing with multi-physics problemsParallel Computational implementation.Generalized form with divergence, curl and gradientPotential to apply in the fast growing grid computing.A variety of interesting physical phenomena and the efficacy of the computational implementation
Thanks
Any Questions?