Known Knowns, Known Unknowns, Unknown Unknowns: The predicament of evidence-based policy
Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells)...
Transcript of Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells)...
![Page 1: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/1.jpg)
Reza Rooholamini, Ph.D.Director
Enterprise SolutionsDell Computer Corp.
http://www.dell.com/clustering
Dell High-Performance Computing Clusters and
Reservoir Simulation Research at UT Austin
![Page 2: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/2.jpg)
Enterprise Solutions2
Cos
t/C
ompl
exit
yProduct Maturity Life Cycle in the Open Systems Market
Proprietary StandardizationFully
Standardized
4P servers1/2P serversAppliance Servers
Network Attached Storage
Project based SANs
Heterogeneous SANs
Simplicity/Volume/Choice
Direct Attached Storage
RISC systems
8P servers
WorkstationDesktops
HPC Clusters
Grids
![Page 3: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/3.jpg)
Enterprise Solutions3
Our Vision
• Customers define our success: Begin with the customer. End with the customer
• Provide the best price/performance solutions to our customers inHPC
• Promote standardization to provide choice, lower cost of ownership, and simplicity in HPC solutions
• Evangelize new HPC technologies and selectively adopt the relevant ones for “productization”
• Derive the requirements for products by focusing on applications
• Provide a total solution: Hardware, software and services
• Partner with “best of class” in HPC
![Page 4: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/4.jpg)
Enterprise Solutions4
Infiniband
Dell PowerEdge Servers (IA32 & IA64)
Parallel Benchmarks (NAS, HINT, Linpack…) Parallel Benchmarks (NAS, HINT, Linpack…) and Parallel Applicationsand Parallel Applications
VIA
Fast Ethernet
TCP
PlatformPlatformPlatform
InterconnectInterconnectInterconnect
ProtocolProtocolProtocol
OSOSOS
MiddlewareMiddlewareMiddleware
BenchmarkBenchmarkBenchmark
Gigabit Ethernet Myrinet
GM
Linux Windows
MPI/Pro PVMMPICH MVICH
Quadrics
Elan
Building Block Approach
![Page 5: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/5.jpg)
Enterprise Solutions5
Dell and UT Austin
• Dell is sponsoring research in reservoir simulation at the Department of Petroleum and Geosystems Engineering
• Dr. Kamy Sepehrnoori is collaborating with Dell’s HPCC team on performance studies, paper publications, and parallel simulator development
• Dell HPCC team includes graduates from Dr. Sepehrnoori’s group specialized in Petroleum Engineering
• Dell has participated in Reservoir Simulation JIP (Joint Industry Project) in the past, and is planning to attend the upcoming meeting
• Dr. Sepehrnoori has access to Dell HPC lab for running large simulations, and is provided with hardware for development, testing, and performance studies of his program
![Page 6: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/6.jpg)
CPGE
A Performance Study of Parallel Reservoir Simulation on
HPC Clusters
Baris Guler
Tau Leng
Victor Mashayekhi
Reza Rooholamini
Dell Computer CorporationDell Computer Corporation
Kamy Sepehrnoori
Center for Petroleum and Geosystems EngineeringCenter for Petroleum and Geosystems Engineering
The University of Texas at AustinThe University of Texas at Austin
![Page 7: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/7.jpg)
CPGE
Outline
Background
Software/Hardware Description
Compositional Reservoir simulation on HPCs
Results
Summary
Future Work
![Page 8: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/8.jpg)
CPGE
Reservoir Simulation Application
Reservoir Forecasting
Reservoir Performance optimization
Sensitivity Analysis
History Matching
Risk Assessment through Stochastic Simulation
Assessment of Uncertainity in Forecasting
Value of Information Studies
Reservoir Management
![Page 9: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/9.jpg)
CPGE
Reservoir Simulation Steps
Data Input/Model Initialization
Do Time Step Computation
Solution of Non-Linear Partial Differential Equation
• Discretization
• Linearization and Newtonian Iteration
Solution using Direct or Iterative Solvers
Test for Convergence of Solution
• Data Output/Graphics
• Time-Step Increment
End of Simulation Study
Results Processing/Interpretation
![Page 10: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/10.jpg)
CPGE
Reservoir Simulation Hardware
Mainframes
Supercomputers
RISC Workstations
PCs/Workstations
HPCs
1960 2000
MPPs
![Page 11: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/11.jpg)
CPGE
Benefits of Parallel Processing
Turn-around time
Large-scale simulations
Cost
![Page 12: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/12.jpg)
CPGE
Massively Parallel Computers
High Performance Computing Clusters
Parallel Processing
![Page 13: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/13.jpg)
CPGE
Benefits of Clusters
Scalability
High Performance Computing
Low Cost
Availability
![Page 14: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/14.jpg)
CPGE
Computational Mode
Distributed processing
Parallel processing
![Page 15: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/15.jpg)
CPGE
Database Post Processing
Distributed Processing
Batch Queuing System to Simulation Program
D1 D2 D3 Dn
n >> m
P1 P2 P3 Pm
User
Input Generator
![Page 16: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/16.jpg)
CPGE
Cluster Simulation System
InputInputDataData
FS 1FS 1
. . .
. . .
FS 2FS 2
FS 3FS 3
FS mFS mClu
ster
Sch
edul
erC
lust
er S
ched
ulerDS 1DS 1
. . .
. . .
DS 2DS 2
DS 3DS 3
DS nDS n
Dat
a G
ener
ator
Dat
a G
ener
ator
Project AdvisorProject Advisor
UserUserInputInput
Arc
hive
rA
rchi
ver
OutputOutput
Post
Post
-- Pro
cess
orPr
oces
sor
![Page 17: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/17.jpg)
CPGE
![Page 18: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/18.jpg)
CPGE
Parallel Processing
CPU-1CPU-1
RESERVOIR
FD
FD & DDCPU-1CPU-1 CPU-2CPU-2 CPU-3CPU-3
CPU-4CPU-4 CPU-5CPU-5 CPU-6CPU-6
RESERVOIRRESERVOIR
![Page 19: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/19.jpg)
CPGE
Domain Decomposition
Fundamental strategy for grid-based parallel simulation.
Example:10 x 15 grid 6 processors
Ghost Layers CommunicationGhost Layers CommunicationGhost Layers CreationGhost Layers Creation
![Page 20: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/20.jpg)
CPGE
Performance Issues in Parallel Processing
Software DesignAlgorithmParallelizationProgramming practiceLoad Balancing
![Page 21: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/21.jpg)
CPGE
Performance Issues in Parallel Processing
Hardware Configuration
CPU
Cache
Memory subsystem
Front Side Bus
I/O bandwidth
Interconnect
![Page 22: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/22.jpg)
CPGE
Hardware - Interconnect
4385Dolphin
4.5330Quadrics
6-8500Infiniband 4x
6-7225Myrinet
7.5110Giganet
17080Gigabit Ethernet
1709.0Fast Ethernet
Latency(ms)Speed(MBps)Type
![Page 23: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/23.jpg)
CPGE
CPGE-1(Ararat)
12 Nodes / 16 Processors1.0 GHz Intel Pentium III Xeon processors256 MB of memory Diskless configuration100 Mbps switched Fast Ethernet and Giganet interconnects
![Page 24: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/24.jpg)
CPGE
TACC-1(Tejas)• 32 Nodes / 64 Processors• 1.0 GHz Intel Pentium III processors• 1 GB of memory/processor• 225 MBps Myrinet-2000 interconnect
![Page 25: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/25.jpg)
CPGE
Parallel Reservoir SimulatorsChevron-Texaco
Conoco-Phillips
Exxon-Mobil
IFP and Beicip-Franlab
Landmark Graphics Corporation
Schlumberger-Geoquest
Saudi Aramco
UT CPGE, UT CSM
Note : 93 clusters in Top500 supercomputer sites, 23 in Oil and Gas sector.
![Page 26: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/26.jpg)
CPGE
Compositional Reservoir Simulation on HPCs
![Page 27: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/27.jpg)
CPGE
Project Objectives
Develop a general purpose adaptive simulator (GPAS) capable of:
modeling of complex physical processes including EOS compositional, chemical, black-oil and thermal
high resolution studies on supercomputers and high-performance cluster
![Page 28: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/28.jpg)
CPGE
HPC Initiatives
Evaluate and compare performance of different
cluster systems
Test and analyze performance of different parallel
simulators
Identify areas of improvement in parallel algorithm
design and cluster setup for optimal parallel
reservoir simulation
![Page 29: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/29.jpg)
CPGE
Summary of Clusters
IBM SP Switch22GB4x16=64
1300Power4
TACC-2 (Longhorn)
Myrinet512MB32x2=64
1000Pentium IIITACC-1 (Tejas)
Myrinet, Gigabit,
Fast Ethernet1GB64x2=1282400Intel Xeon DP
DELL-2 (PE 2650)
Myrinet, Gigabit,
Fast Ethernet512MB
16x2=321000
Pentium IIIDELL-1 (PE 1550)
Fast Ethernet256MB
8x1+4x2=161000
Pentium III XeonCPGE-1 (Ararat)
Fast Ethernet256MB
8x2=16400
Pentium II XeonCPGE-1 (Rocky)
Fast Ethernet384MB
16x1=16300
Pentium IICPGE-1 (Fuji)
InterconnectMemory per CPUCPUsCPU Speed
(MHz)CPU TypeCluster
![Page 30: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/30.jpg)
CPGE
Parallel Simulators Tested
GPASGPAS
VIP (2003r4)VIP (2003r4)
![Page 31: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/31.jpg)
CPGE
CPGE Simulator (GPAS)
EOS Compositional
Peng-Robinson EOS
Fully Implicit
PETSc Linear Solvers
Parallel (IPARS Framework)
![Page 32: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/32.jpg)
CPGE
Performance Results
![Page 33: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/33.jpg)
CPGE
Base Benchmark Problem
Compositional model3-component Peng-Robinson EOS Dry-gas cycling processReservoir size: 800 x 11200 x 160 ft, homogeneous2 wells, 1 Injector, 1 producerGrids: 16 x 224 x 8 (28,672 cells)Unknowns : 229,376100 days of gas injectionOne dimensional domain decomposition
![Page 34: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/34.jpg)
CPGE
SingleSingle--Processors’ Execution Processors’ Execution Times(GPASTimes(GPAS))Base Benchmark Problem
180.186
306.38
309.3
313.3
615.2
1030.3
0 200 400 600 800 1000 1200
Dell-PE2650 withIntel Xeon DP
2.4GHz
TACC-Tejas withPentium III
1.0Ghz
Ararat withPentium III Xeon
1.0GHz
PowerEdge 1550with Pentium III
1.0GHz
Rocky withPentium II Xeon
400MHz
Fuji with PentiumII 300MHz
Execution Time [sec]
![Page 35: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/35.jpg)
CPGE
MultiMulti--Processors’ Execution Processors’ Execution Times(GPASTimes(GPAS))Base Benchmark Problem
1
10
100
1000
10000
0 4 8 12 16
Number of Processors
Exec
utio
n Ti
me
(sec
onds
)FujiRockyAraratPE 1550PE 2650TejasLonghorn
![Page 36: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/36.jpg)
CPGE
MultiMulti--Processors’ Processors’ Speedups(GPASSpeedups(GPAS))Base Benchmark Problem
0
4
8
12
16
20
24
28
32
0 4 8 12 16 20 24 28 32
Number of Processors
Sppe
dup
Fuji(FE)Rocky(FE)Ararat(FE)PE 1550(FE)PE 2650(FE)Tejas(My)Longhorn(*)Ideal
![Page 37: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/37.jpg)
CPGE
Comparison of MPIComparison of MPI--Interconnects (GPAS)Interconnects (GPAS)Base Benchmark Problem
DELL PE 2650 (Single processor/node)
0
4
8
12
16
20
24
28
32
0 4 8 12 16 20 24 28 32
Number of Processors
Spee
dup
MPICH-GIGABIT MPICH-GM - MYRINET MPI/PRO-GIGABITMPICH-FE Ideal
![Page 38: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/38.jpg)
CPGE
Constant Problem Size per Constant Problem Size per Processor(GPASProcessor(GPAS))
0
200
400
600
800
19200,1CPU 38400, 2CPUs 76800, 4CPUs 153600, 8CPUs 307200,16CPUs
614400,32CPUs
Grid Dimensions, Number of CPUs
Exec
utio
n Ti
me
[sec
]
Fuji Rocky Ararat Tejas
![Page 39: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/39.jpg)
CPGE
Modified Benchmark Problem
Compositional model3-component Peng-Robinson EOS Dry-gas cycling processReservoir size: 7.3 x 24.2 x .1 miles Grids: 77 x 256 x 10 (197,120 cells)Unknowns : 1.57 millionAnisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells, 54 Injectors , 24 producers, staggered line driveInjectors and Producers are completed fully100 days of gas injectionOne dimensional domain decomposition
![Page 40: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/40.jpg)
CPGE
MultiMulti--Processors’ Execution Processors’ Execution Times(GPASTimes(GPAS))Modified Benchmark ProblemModified Benchmark Problem
DELL PE 2650
10
100
1000
10000
0 8 16 24 32 40 48 56 64
Number of Processors
Exe
cutio
n Ti
me
(Sec
onds
)
GBit-SINGLE My-SINGLE FE-SINGLE My-DUAL
![Page 41: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/41.jpg)
CPGE
MultiMulti--Processors’ Processors’ Speedups(GPASSpeedups(GPAS))Modified Benchmark Problem
DELL PE 2650
0
8
16
24
32
40
48
56
64
72
0 8 16 24 32 40 48 56 64Number of Processors
Spee
dup
GIGABIT-SINGLE MYRINET-SINGLE FAST ETH-SINGLEMYRINET-DUAL Ideal
![Page 42: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/42.jpg)
CPGE
Commercial Parallel Simulator
![Page 43: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/43.jpg)
CPGE
REMARKS
Our goal was to run the simulators in parallel mode and evaluate their performance for typical casesOur goal was to analyze the different issues involved in using the simulators in parallel and approaches to improved performance and designWe did not
Tune simulators for optimum performanceCompare or match material balance errors of the simulator runs
![Page 44: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/44.jpg)
CPGE
Benchmark Problem for VIP
Compositional model – Modified SPE3 comparison project9-component Peng Robinson EOS Gas condensate with gas cycling processReservoir size: 10 miles x 4 miles x 160ftGrids: 180 x 72 x 4 (51,840 cells)1 million unknownsFlow barriers present (using Transmissibility modifiers)20 wells, 10 Injectors , 10 producers10 years of cycling followed by 5 years of production
![Page 45: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/45.jpg)
CPGE
Multi-Processors’ PerformanceVIP
![Page 46: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/46.jpg)
CPGE
MultiMulti--Processors’ Execution Processors’ Execution Times(VIPTimes(VIP))MODIFIED SPE3 COMPARISON PROBLEM
02000400060008000
1000012000
0 4 8 12 16
Number of Processors
Elap
sed
Tim
e (s
ec)
Fuji Rocky
![Page 47: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/47.jpg)
CPGE
MultiMulti--Processors’ Processors’ Speedups(VIPSpeedups(VIP))MODIFIED SPE3 COMPARISON PROBLEM
0
4
8
12
16
0 4 8 12 16Processors
Spee
dup
Fuji Rocky Ideal
![Page 48: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/48.jpg)
CPGE
Constant Problem Size per Constant Problem Size per Processor(VIPProcessor(VIP))MODIFIED SPE3 COMPARISON PROBLEM
0100020003000400050006000700080009000
10000
25920, 1CPU 51480, 2CPUs 103680, 4CPUs 207360, 8CPUs 414720,16CPUsNumber of Cells, Number of CPUs
Exec
utio
n Ti
me
[sec
]
Fuji Rocky
![Page 49: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/49.jpg)
CPGE
Million Cell Commercial Benchmark Problem for VIP
IMPES scheme7-component Peng Robinson EOS Grid: 100 x 100 x 100 (1 Million cells)16 million unknownsStochastically characterized data field11 wells49 Year run
![Page 50: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/50.jpg)
CPGE
Performance Speedups - VIPMILLION GRIDBLOCK PROBLEM
DELL PE 2650
0
8
16
24
32
40
48
56
64
0 8 16 24 32 40 48 56 64Number of Processors
Spee
dup
VIP(*)
Ideal
![Page 51: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/51.jpg)
CPGE
Summary
Tested GPAS and analyzed performance on new hardwareBenchmarked performance of new clustersCompared performance of different interconnects and MPI librariesTested commercial reservoir simulator VIP in parallel mode
![Page 52: Dell High-Performance Computing Clusters and Reservoir ......Grids: 77 x 256 x 10 (197,120 cells) Unknowns : 1.57 million Anisotropic, Layered Permeability with Kv/Kh = 0.1 88 wells,](https://reader033.fdocuments.in/reader033/viewer/2022060405/5f0f2c927e708231d442da79/html5/thumbnails/52.jpg)
CPGE
Acknowledgements
US Department of Energy
Reservoir Simulation Joint Industry Project Members
Dell Computer Corporation