High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM...
Transcript of High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM...
![Page 1: High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,](https://reader030.fdocuments.in/reader030/viewer/2022040916/5e8e70ceaee3284d775b25f4/html5/thumbnails/1.jpg)
IBM ATS Deep Computing
© 2007 IBM Corporation
High Performance IO HPC Workshop – University of KentuckyMay 9, 2007 – May 10, 2007
Andrew Komornicki, Ph. D.Balaji Veeraraghavan, Ph. D.
![Page 2: High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,](https://reader030.fdocuments.in/reader030/viewer/2022040916/5e8e70ceaee3284d775b25f4/html5/thumbnails/2.jpg)
IBM ATS Deep Computing
© 2007 IBM Corporation
Agenda
Introduction
General IO performance
Results of some small tests.
Modular IO libraries, Linux and AIX
![Page 3: High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,](https://reader030.fdocuments.in/reader030/viewer/2022040916/5e8e70ceaee3284d775b25f4/html5/thumbnails/3.jpg)
IBM ATS Deep Computing
© 2007 IBM Corporation
I/O Optimization
Analyze the IO pattern
Determine optimization method
Optimize in user space
Minimize source code changes
Possibly relink with libtkio.so
![Page 4: High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,](https://reader030.fdocuments.in/reader030/viewer/2022040916/5e8e70ceaee3284d775b25f4/html5/thumbnails/4.jpg)
IBM ATS Deep Computing
© 2007 IBM Corporation
General I/O Performance
C: Do not use fopen(), fread(), or fwrite(); • These are inefficient due to small (4KB) IO blocks and extra
memory copies.Use instead: • POSIX open(), read(), write() • Direct (raw) IO will eliminate an additional memory copy
FORTRAN: Use unformatted IO
![Page 5: High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,](https://reader030.fdocuments.in/reader030/viewer/2022040916/5e8e70ceaee3284d775b25f4/html5/thumbnails/5.jpg)
IBM ATS Deep Computing
© 2007 IBM Corporation
Asynchronous IO, an example
Non Blocking IO aio_read(), aio_write(), aio_return();
Completion Notification Polling with aio_error(); Block until complete with aio_suspend():
Cancellation of IO requests aio_cancel();
Large File enabled Removes the 2GB file size limitation
POSIX conforming
![Page 6: High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,](https://reader030.fdocuments.in/reader030/viewer/2022040916/5e8e70ceaee3284d775b25f4/html5/thumbnails/6.jpg)
IBM ATS Deep Computing
© 2007 IBM Corporation
Results of Bonnie IO test
Run on Blade system in San Mateo Lab
System Memory, 5 Gbytes
File systems, ext2, and ext3
All tests done in four stages: Writing with putc()...done
Rewriting...done
Writing intelligently...done
Reading with getc()...done
Reading intelligently... done
![Page 7: High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,](https://reader030.fdocuments.in/reader030/viewer/2022040916/5e8e70ceaee3284d775b25f4/html5/thumbnails/7.jpg)
IBM ATS Deep Computing
© 2007 IBM Corporation
Results of Bonnie IO test, Block IO performance
Size (MB) Write (Kbytes/sec) Read(Kbytes/sec)
__________________________________________
2000 841,524 2,233,282
4000 83,237 1,658,013
8000 56,599 50,974
16000 49,656 50,677
![Page 8: High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,](https://reader030.fdocuments.in/reader030/viewer/2022040916/5e8e70ceaee3284d775b25f4/html5/thumbnails/8.jpg)
IBM ATS Deep Computing
© 2007 IBM Corporation
Results of Bonnie IO test
Results for ext2 file system, time in seconds
Size (MB) User System Elapsed
_________________________________
2000 48.7 10.8 84.3
4000 97.6 23.9 252.6
8000 194.9 56.4 1009.2
16000 388.4 111.8 2088.3
![Page 9: High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,](https://reader030.fdocuments.in/reader030/viewer/2022040916/5e8e70ceaee3284d775b25f4/html5/thumbnails/9.jpg)
IBM ATS Deep Computing
© 2007 IBM Corporation
Results of Bonnie IO test
Results for ext3 file system, time in seconds
Size (MB) User System Elapsed
______________________________________
2000 48.7 19.2 96.3
4000 97.7 45.5 265.8
8000 194.6 90.1 1016.9
16000 396.9 201.8 2058.3
![Page 10: High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,](https://reader030.fdocuments.in/reader030/viewer/2022040916/5e8e70ceaee3284d775b25f4/html5/thumbnails/10.jpg)
IBM ATS Deep Computing
© 2007 IBM Corporation
Modular I/O (MIO)Modular I/O (MIO)
Familiar and flexible runtime interfaceMIO modulesmiotracepf
MIO available on both Linux and AIX
![Page 11: High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,](https://reader030.fdocuments.in/reader030/viewer/2022040916/5e8e70ceaee3284d775b25f4/html5/thumbnails/11.jpg)
IBM ATS Deep Computing
© 2007 IBM Corporation
MIO user code interface
open MIO_open
read MIO_read
write MIO_write
close MIO_close
lseek MIO_lseek
fcntl MIO_fcntl
ftruncate MIO_ftruncate
![Page 12: High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,](https://reader030.fdocuments.in/reader030/viewer/2022040916/5e8e70ceaee3284d775b25f4/html5/thumbnails/12.jpg)
IBM ATS Deep Computing
© 2007 IBM Corporation
MIO run time interfaceMIO run time interface
MIO_STATS="file name"MIO_FILES=" *.dat* [trace|pf ] *.inp [aix]"MIO_DEBUG="ALL"MIO_DEFAULTS="trace/mbytes , pf/cache=10m“
![Page 13: High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,](https://reader030.fdocuments.in/reader030/viewer/2022040916/5e8e70ceaee3284d775b25f4/html5/thumbnails/13.jpg)
IBM ATS Deep Computing
© 2007 IBM Corporation
trace moduletrace modulesummary of file activitybinary events filelow cpu overheadtypical options
/stats/mbytes /gbytes /tbytes/events=mio.evt
![Page 14: High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,](https://reader030.fdocuments.in/reader030/viewer/2022040916/5e8e70ceaee3284d775b25f4/html5/thumbnails/14.jpg)
IBM ATS Deep Computing
© 2007 IBM Corporation
pf module
User selectable cache size
User selectable page size
User selectable prefetch depth
Direct or system buffered IO
Global or private cache
Usage summary
![Page 15: High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,](https://reader030.fdocuments.in/reader030/viewer/2022040916/5e8e70ceaee3284d775b25f4/html5/thumbnails/15.jpg)
IBM ATS Deep Computing
© 2007 IBM Corporation
pf modulepf moduledetects sequential I/Ouser memory bufferingoptions
/global/cache_size=10m/page_size=1m/prefetch=1/stride=1/direct/stats
![Page 16: High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,](https://reader030.fdocuments.in/reader030/viewer/2022040916/5e8e70ceaee3284d775b25f4/html5/thumbnails/16.jpg)
IBM ATS Deep Computing
© 2007 IBM Corporation
Relink with libtkio.a
libtkio.a has shared object memberstkio.so 32 bit and 64 bit
Entry points for• open,open64,close,read,write,lseek,lseek64• fcntl,ffinfo,fstat,fstat64,fstatfs,fsync• ftruncate,ftruncate64• unlink,aio_...
![Page 17: High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,](https://reader030.fdocuments.in/reader030/viewer/2022040916/5e8e70ceaee3284d775b25f4/html5/thumbnails/17.jpg)
IBM ATS Deep Computing
© 2007 IBM Corporation
Default tkio behavior
Uses dlopen and dlsym for runtime linking
……libc(shr.o) fsyncfsynclibc(shr.o) lseek64lseek64libc(shr.o) writewritelibc(shr.o) readreadlibc(shr.o) closecloselibc(shr.o) open64open64
callstkio entry
![Page 18: High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,](https://reader030.fdocuments.in/reader030/viewer/2022040916/5e8e70ceaee3284d775b25f4/html5/thumbnails/18.jpg)
IBM ATS Deep Computing
© 2007 IBM Corporation
tkio runtime interface
setenv TKIO_ALTLIB so_name/print/abort
export TKIO_ALTLIB=so_name/print/abort
so_name is name of shared library• Either name.so or libname.a(name.so)
tkio calls function in so_name that returns a structure filled with I/O entry points to replace default entry points
/print option outputs a print to stderr indicating success of load
/abort issues exit(-1) if load is not successfull
![Page 19: High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,](https://reader030.fdocuments.in/reader030/viewer/2022040916/5e8e70ceaee3284d775b25f4/html5/thumbnails/19.jpg)
IBM ATS Deep Computing
© 2007 IBM Corporation
tkio using MIOsetenv TKIO_ALTLIB get_mio_ptrs_64.so
…libmio(mio.o) MIO_fsyncFsynclibmio(mio.o) MIO_lseek64Lseek64libmio(mio.o) MIO_writeWritelibmio(mio.o) MIO_readReadlibmio(mio.o) MIO_closeCloselibmio(mio.o) MIO_open64Open64
Calls tkio entry
![Page 20: High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,](https://reader030.fdocuments.in/reader030/viewer/2022040916/5e8e70ceaee3284d775b25f4/html5/thumbnails/20.jpg)
IBM ATS Deep Computing
© 2007 IBM Corporation
kernel
Application libclibtkio
Fortran I/O
Demonstration only
open64writereadlseek64close
->open64->write->read->lseek64->close
stdiofopenfrwritefreadfclose
libmio
->MIO_open64->MIO_write->MIO_read->MIO_lseek64->MIO_close
X
![Page 21: High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,](https://reader030.fdocuments.in/reader030/viewer/2022040916/5e8e70ceaee3284d775b25f4/html5/thumbnails/21.jpg)
IBM ATS Deep Computing
© 2007 IBM Corporation
kernel
libclibtkio
open64writereadlseek64close
->open64->write->read->lseek64->close
libmio
->MIO_open64->MIO_write->MIO_read->MIO_lseek64->MIO_close t
race
pf
aix
![Page 22: High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,](https://reader030.fdocuments.in/reader030/viewer/2022040916/5e8e70ceaee3284d775b25f4/html5/thumbnails/22.jpg)
IBM ATS Deep Computing
© 2007 IBM Corporation
System buffered Data MovementSystem buffered Data Movement
user space
kernel
256kb
system buffers
MIO space
![Page 23: High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,](https://reader030.fdocuments.in/reader030/viewer/2022040916/5e8e70ceaee3284d775b25f4/html5/thumbnails/23.jpg)
pf cached Data Movementpf cached Data Movement
user space
kernel
256kb
5 x 2mb
system buffers
MIO space
![Page 24: High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,](https://reader030.fdocuments.in/reader030/viewer/2022040916/5e8e70ceaee3284d775b25f4/html5/thumbnails/24.jpg)
O_DIRECT Data MovementO_DIRECT Data Movement
user space
kernel
O_DIRECT
256kb
5 x 2mb
system buffers
MIO space
![Page 25: High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,](https://reader030.fdocuments.in/reader030/viewer/2022040916/5e8e70ceaee3284d775b25f4/html5/thumbnails/25.jpg)
Asynchronous Data MovementAsynchronous Data Movement
user space
kernel
O_DIRECT
256kb
5 x 2mb
system buffers
MIO space
![Page 26: High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,](https://reader030.fdocuments.in/reader030/viewer/2022040916/5e8e70ceaee3284d775b25f4/html5/thumbnails/26.jpg)
IBM ATS Deep Computing
© 2007 IBM Corporation
Trace close : program <-> pf : /bmwfs/cdh108.T20536_13.SCR300 :(281946/2162.61)=130.37 mbytes/s
current size=0 max_size=16277mode =0777 sector size=4096oflags =0x302=RDWR CREAT TRUNCopen 1 0.01write 478193 462.10 59774 59774 131072 131072read 1777376 1700.48 222172 222172 131072 131072seek 911572 2.83fcntl 3 0.00trunc 16 0.40close 1 0.03size 127787
MSC.NASTRANMSC.NASTRANtracetrace output from output from program <program <-->pf>pf
Min/MaxRequest sizein bytes
Mbytes requestedand Mbytes delivered
Number ofoccurances
![Page 27: High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,](https://reader030.fdocuments.in/reader030/viewer/2022040916/5e8e70ceaee3284d775b25f4/html5/thumbnails/27.jpg)
IBM ATS Deep Computing
© 2007 IBM Corporation
Trace close : pf <-> aix : /bmwfs/cdh108.T20536_13.SCR300 : (276645/1460.73)=189.39 mbytes/s
current size=0 max_size=16276mode =0777 sector size=4096oflags =0x8000302=RDWR CREAT TRUNC DIRECTopen 1 0.01write 4382 154.86 684 684 131072 2097152awrite 33390 1.42 58491 58491 131072 2097152suspend 33390 240.00 242.27 mbytes/sread 5178 272.71 10354 10354 1048576 2097152aread 103560 5.70 207115 207115 524288 2097152suspend 103560 786.04 261.59 mbytes/sseek 136950 0.00fcntl 3 0.00trunc 16 0.40close 1 0.00size 11013pages 138477
MSC.NASTRAN MSC.NASTRAN tracetrace outputoutput
![Page 28: High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,](https://reader030.fdocuments.in/reader030/viewer/2022040916/5e8e70ceaee3284d775b25f4/html5/thumbnails/28.jpg)
IBM ATS Deep Computing
© 2007 IBM Corporation
pf close for /bmwfs/cdh108.T20536_13.SCR300global cache 0: 150 pages of 2097152 bytes29739/29749 pages not preread for write138316/139754 prefetches : prefetch=3
29576 write behinds478193 writes1777376 reads
page writes 37772/33124mbytes transferredprogram --> 59774 --> pf --> 59176 --> aixprogram <-- 222172 <-- pf <-- 217469 <-- aix
MSC.NASTRAN MSC.NASTRAN pfpf outputoutput
![Page 29: High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,](https://reader030.fdocuments.in/reader030/viewer/2022040916/5e8e70ceaee3284d775b25f4/html5/thumbnails/29.jpg)
IBM ATS Deep Computing
© 2007 IBM Corporation
time ( seconds )
file position ( bytes )
DataView file activity plotDataView file activity plot
![Page 30: High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,](https://reader030.fdocuments.in/reader030/viewer/2022040916/5e8e70ceaee3284d775b25f4/html5/thumbnails/30.jpg)
IBM ATS Deep Computing
© 2007 IBM Corporation
time ( seconds )
file position ( bytes )
DataView file activity plotDataView file activity plot
![Page 31: High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,](https://reader030.fdocuments.in/reader030/viewer/2022040916/5e8e70ceaee3284d775b25f4/html5/thumbnails/31.jpg)
IBM ATS Deep Computing
© 2007 IBM Corporation
time ( seconds )
file position ( bytes )
suspend time
hidden time
queuing time
Asynchronous I/O plottingAsynchronous I/O plotting
![Page 32: High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,](https://reader030.fdocuments.in/reader030/viewer/2022040916/5e8e70ceaee3284d775b25f4/html5/thumbnails/32.jpg)
IBM ATS Deep Computing
© 2007 IBM Corporation
time ( seconds )
file position ( bytes )
cache page activitycache page activity
![Page 33: High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,](https://reader030.fdocuments.in/reader030/viewer/2022040916/5e8e70ceaee3284d775b25f4/html5/thumbnails/33.jpg)
IBM ATS Deep Computing
© 2007 IBM Corporation
MSC.Nastran performance gainsMSC.Nastran performance gains
16 cpu 32GB NH2 node 2.2M dof, 767GB I/O, 8 copies2GB memory per copy
114MB/sec 198MB/sec
8 SSA, 16 loops, 4 disk/loop
![Page 34: High Performance IO HPC Workshop – University of Kentucky ......IBM ATS Deep Computing © 2007 IBM Corporation High Performance IO HPC Workshop – University of Kentucky May 9,](https://reader030.fdocuments.in/reader030/viewer/2022040916/5e8e70ceaee3284d775b25f4/html5/thumbnails/34.jpg)
IBM ATS Deep Computing
© 2007 IBM Corporation
MIO Summary
Demonstrated performance gains
Simple to implement
Flexible run time interface
Delivered as a shared object library
Contact: [email protected]