Disk I/O Performance Emphasizing Caching and XFC.
-
Upload
ariel-woods -
Category
Documents
-
view
229 -
download
2
Transcript of Disk I/O Performance Emphasizing Caching and XFC.
Disk I/O PerformanceDisk I/O Performance
Emphasizing Caching and XFCEmphasizing Caching and XFC
Ian PercivalIan PercivalIan PercivalIan Percival XFC Project Leader OpenVMS Engineering [email protected]
Application I/O PerformanceApplication I/O PerformanceApplication I/O PerformanceApplication I/O Performance
Can you improve your I/O performance? Best performance comes from not doing I/O at all,
otherwise get as “close” to the application as you can. Application level considerations. Operating System Constraints and Considerations. Hardware Constraints and considerations.
Hardware ConsiderationsHardware ConsiderationsHardware ConsiderationsHardware Considerations– Disk Types and caches
DecRam 15000, 10000, 7200 rpm, solid state Caches disabled on local SCSI
– Disk Controllers and caches Writethrough, Writeback, Memory
– RAID level– Interconnects
Latency and throughput
– CPU and I/O configuration SMP NUMA
Software ConfigurationSoftware ConfigurationSoftware ConfigurationSoftware Configuration File system
– Fragmentation RMS
– Tuning– Local and Global Buffers
XFC Raid Fastpath MSCP
Tools for analysisTools for analysisTools for analysisTools for analysis
Application itself– Transaction monitor, etc
Availability Manager SHOW and MONITOR SDA Third Party products
Minimize I/O in the first placeMinimize I/O in the first placeMinimize I/O in the first placeMinimize I/O in the first place
Good application design Application knowledge of “hot files” should use
global and local buffers. Try to satisfy I/O as close to the application as
possible, by using caches – software and hardware.
Software HierarchySoftware HierarchySoftware HierarchySoftware Hierarchy
Disk DriverDisk Driver
RMSRMS
File System (XQP)File System (XQP)
ApplicationApplication
eXtended File CacheeXtended File CacheeXtended File CacheeXtended File Cache
Replaces VCC in V7.3 Scales - 64 bit cache can be very large No closed file limit I/Os larger than 35 blocks Readahead where appropriate
Sample Commercial ProgramSample Commercial ProgramSample Commercial ProgramSample Commercial Program
Real world case study I/O intensive Financial Program ES40 4 CPUs Local 10,000 rpm SCSI, 2 volume
set 65% reads. Approximately 7.5 Million I/Os. Clustered, but only a Single Node is performing
the job.
Elapsed TimesElapsed TimesElapsed TimesElapsed Times No Cache or extra RMS buffers
– 43 Hours 28 Mins 45 Secs VCC (the old data cache)
– 8 Hours 15 Mins 22 Secs Out of the box V7.3 using XFC
– 7 Hours 11 Mins 16 Secs Use RMS Global Buffers on 10 “hot files” with no cache
– 12 Hours 4 Mins 55 Secs XFC and RMS Global buffers
– 7 Hours 11 Mins 55 Secs
RMS Global Buffers and/or XFC?RMS Global Buffers and/or XFC?RMS Global Buffers and/or XFC?RMS Global Buffers and/or XFC? RMS Global buffers WILL cache files open for writeshare
in a cluster XFC will not in V7.3 Obviously RMS buffers require knowledge of IO to apply
them effectively. Can use the SET FILE/GLOBAL_BUFFER=n command to enable them once you know which files are “hot”.
XFC is S2 Space based and has no effective limit to its size other than physical memory available
RMS Global buffers are currently limited to a max value of 32767 buffers per file.
RMS Global Buffers AND XFC make a RMS Global Buffers AND XFC make a good pairgood pairRMS Global Buffers AND XFC make a RMS Global Buffers AND XFC make a good pairgood pair
As RMS is closer to the User/App it always makes sense to use it.
Must use global buffers for files open for shared write in a cluster – RIGHTSLIST.DAT is a good example here on a common system disk.
XFC will cache as much as it can once RMS passes I/O to it
Hot Files and Data AccessesHot Files and Data AccessesHot Files and Data AccessesHot Files and Data Accesses
All Disk I/O flows through XFC Enhancement to DCL SHOW MEM
– SHOW MEMORY/CACHE=
– (VOLUME=PICKLE, TOPQIO=20, OPEN, CLOSED)
– Limitation: Only works for Virtual I/O Can also use various third party products
HotFiles CommandHotFiles CommandHotFiles CommandHotFiles Command
$ show memory/cache=(volume=iss$,topqio=2,open,closed) System Memory Resources on 29-SEP-2000 10:09:56.86
Extended File Cache Top QIO File Statistics:
_$9$DKB500: (DISK$ISS_V1), Caching mode is VIOC Compatible
_$9$DKB500:[iSS$ROOT.iSS$.APPFILES6]COBEGOP.PLO;72 (open) Caching is enabled, active caching mode is Write Through Allocated pages 7664 Total QIOs 2650066 Read hits 2122691 Virtual reads 2133065 Virtual writes 517001 Hit rate 80 % Read aheads 0 Read throughs 2133065 Write throughs 517001 Read arounds 0 Write arounds 0
I/O and Cache Statistics for the VolumeI/O and Cache Statistics for the VolumeI/O and Cache Statistics for the VolumeI/O and Cache Statistics for the Volume
$show memory/cache=volume=iss$ System Memory Resources on 29-SEP-2000 10:27:23.36
Extended File Cache Volume Statistics:
_$9$DKB500: (DISK$ISS_V1), Caching mode is VIOC Compatible Open files 115 Closed files 110 Files ever opened 1309 Files ever deposed 1084 Allocated pages 143512 Locks acquired 1198 Total QIOs 4425333 Read hits 3211408 Virtual reads 3366252 Virtual writes 1056131 Hit rate 72 % Read aheads 6638 Read throughs 3365919 Write throughs 1056131 Read arounds 333 Write arounds 0 Ave Disk I/O Resp Time incl cache hits (microseconds) 3107
SHOW MEMORY/CACHE - NODEWIDESHOW MEMORY/CACHE - NODEWIDESHOW MEMORY/CACHE - NODEWIDESHOW MEMORY/CACHE - NODEWIDE
show mem/cache
System Memory Resources on 29-SEP-2000 11:06:14.21
Extended File Cache (Time of last reset: 28-SEP-2000 11:29:35.85) Allocated (Mbytes) 3000.00 Maximum size (Mbytes) 5120.00 Free (Mbytes) 976.38 Minimum size (Mbytes) 3000.00 In use (Mbytes) 2023.61 Write hit rate 0% Read hit rate 93% Write I/O count 2731285 Read I/O count 5346002 Write hit count 0 Read hit count 5012771 Writes bypassing cache 0 Reads bypassing cache 337 Vols in Full XFC mode 0 Files cached open 522 Vols in VIOC Compatible mode 3 Files cached closed 478 Vols in No Caching mode 0 Percentage Read I/Os 66% Vols in Perm. No Caching mode 0
SHOW MEMORY/CACHE/FULLSHOW MEMORY/CACHE/FULLSHOW MEMORY/CACHE/FULLSHOW MEMORY/CACHE/FULL I/O Statistics - Distributions (MAX_IO_SIZE: 127) ------------------------------------------------- Transfer Size: Reads Read Hits Writes 1 Block IO: 472510 472446 115923 2 Block IO: 7553 7481 426 3 Block IO: 2068434 2035932 753483 4 Block IO: 199136 172784 324110 5 Block IO: 27315 22481 50520 6 Block IO: 1613335 1475185 1205452 7 Block IO: 109592 107681 155111 8 Block IO: 16076 16041 146 9 Block IO: 748511 620923 117591 10 Block IO: 78 67 165 11 Block IO: 884 879 8 12 Block IO: 7572 7563 3314 13 Block IO: 3333 3329 6 14 Block IO: 56 48 13 15 Block IO: 1487 1486 10 16 Block IO: 8900 7870 2086 17 Block IO: 374 368 9 18 Block IO: 21514 21153 2841 19 Block IO: 60 54 5 20 Block IO: 17 13 0
Per Volume and Per File IO HistogramPer Volume and Per File IO HistogramPer Volume and Per File IO HistogramPer Volume and Per File IO Histogram Not there by default Will provide a utility to enable this
– (not there as standard in V7.3) Per file and per volume Statististics Resets are part of this SET CACHE/RESET
More on Response TimesMore on Response TimesMore on Response TimesMore on Response Times
Many third party products use Little’s Law As does Compaq’s Capacity Planner – Little’s Law can be
manipulated to give:– Response time = average queue length/IO rate
Not too useful for lower I/O Volumes where queues don’t build up. XFC performs an RSCC() call when an I/O enters it. It performs another RSCC() when the I/O is about to complete (User
process is aboutto be awakened again) SMP accuracy limitation RMS buffer hits are not included
How to get more on response timesHow to get more on response timesHow to get more on response timesHow to get more on response times
Use XFC SDA extension for more detail– $ ANALYZE/SYSTEM
– SDA> XFC SHOW VOLUME/BRIEF
– SDA> XFC SHOW VOLUME This is the PER NODE view – need to do this clusterwide
to get the entire picture
XFC SHOW VOLUME/BRIEFXFC SHOW VOLUME/BRIEFXFC SHOW VOLUME/BRIEFXFC SHOW VOLUME/BRIEFSDA> xfc show volume/brief
Summary of XFC Cached Volumes (CVBs)
------------------------------------
Name: DISK$ISS_V2 CVB: FFFFFFFE495FF810 Cached Files: 000000000000000026 (
38)
Name: DISK$ISS_V1 CVB: FFFFFFFE495FFA70 Cached Files: 000000000000000073 (
115)
Name: DISK$X73W_DENBO2 CVB: FFFFFFFE495FFCD0 Cached Files: 000000000000000174 (
372)
Volumes found: 3
XFC SHOW VOLUME 1 of Volume SetXFC SHOW VOLUME 1 of Volume SetXFC SHOW VOLUME 1 of Volume SetXFC SHOW VOLUME 1 of Volume SetCached Open Files: 38
Cached Closed Files: 8
Files Ever Opened: 160
Files Ever Deposed: 114
Pages Allocated: 97270
Total QIOs: 4248036
Read Hit Count: 2372227
Virtual Read Count: 2560160
Virtual Write Count: 1687876
Read Percentage: 60 %
Hit Rate: 55 %
Average Overall I/O response time to this Volume
in microseconds: 4298
Average Cache Hit I/O response time to this Volume
in microseconds: 24
Average Disk I/O response time to this Volume
in microseconds: 9703
Accuracy of I/O resp time: 70 %
Readahead Count: 50
XFC SHOW VOLUME 2 of Volume SetXFC SHOW VOLUME 2 of Volume SetXFC SHOW VOLUME 2 of Volume SetXFC SHOW VOLUME 2 of Volume SetCached Open Files: 115
Cached Closed Files: 110
Files Ever Opened: 1309
Files Ever Deposed: 1084
Pages Allocated: 143517
Total QIOs: 4431899
Read Hit Count: 3217640
Virtual Read Count: 3372484
Virtual Write Count: 1056301
Read Percentage: 76 %
Hit Rate: 72 %
Average Overall I/O response time to this Volume
in microseconds: 3103
Average Cache Hit I/O response time to this Volume
in microseconds: 21
Average Disk I/O response time to this Volume
in microseconds: 11262
Accuracy of I/O resp time: 82 %
Readahead Count: 6720
RMS Global BuffersRMS Global BuffersRMS Global BuffersRMS Global Buffers
SET FILE/STATISTICS to enable monitoring Elinor did a super job by allowing the /SHARE qualifier to let you do
this live. For example:
– SET FILE/STAT/SHARE HALLOWEEN.DAT– MONITOR RMS /ITEM=CACHING/FILE=HALLOWEN.DAT– SET FILE/NOSTAT/SHARE … turns it off
Global Buffers use global sections and hence global pagefile space.
XFC and memory usage.XFC and memory usage.XFC and memory usage.XFC and memory usage.
In general, XFC (or any cache) will be more effective the more main memory you have.
Operates in Dynamic or Permanent or both modes. Permanent memory is assigned by using RESERVED MEMORY. Probably a good idea to reserve some percentage of your memory
for XFC, leaving the remainder dynamic, assuming your system isn’t memory starved.
NUMA support.– RESERVED MEMORY only
XFC SYSGEN ParamersXFC SYSGEN ParamersXFC SYSGEN ParamersXFC SYSGEN Paramers
VCC_MAX_CACHE– Default is -1– (Note not VCC_MAXSIZE – VIOC only)
VCC_MAX_IO_SIZE– Default is 127
(VIOC was hard coded to 34 blocks)
– Histograms should show when you need to increase this
XFC Reserved MemoryXFC Reserved MemoryXFC Reserved MemoryXFC Reserved Memory
SYSMAN registry SYSMAN> RESERVED_MEMORY ADD
VCC$MIN_CACHE_SIZE/ALLOCATE/NOPAGE_TABLES/NOZERO/NOGLOBAL/SIZE=xx
– Size is in units of Mb For NUMA support, you should add /RAD=xx with the above
command, entering it for each RAD you wish to use. XFC Dynamic memory is NOT currently RAD selectable.
SHOW MEMORY with XFC ReservationSHOW MEMORY with XFC ReservationSHOW MEMORY with XFC ReservationSHOW MEMORY with XFC Reservation
$ SHOW MEMORY/RES
Memory Reservations (pages): Group Reserved In Use Type
VCC$MIN_CACHE_SIZE --- 384000 384000 Allocated
Total (2.92 Gbytes reserved) 384000 384000
Don’t Cache unimportant devicesDon’t Cache unimportant devicesDon’t Cache unimportant devicesDon’t Cache unimportant devices
In general, systems are not going to be memory rich. Caching many Tb of disk space with just a few Gb of physical memory may not give high hit rates.
MOUNT/NOCACHE – Disables XQP caches as well as VCC/XFC
For individual control– SET FILE/CACHE=NO_CACHING X.X
Extract of DIR/FULL commandExtract of DIR/FULL commandExtract of DIR/FULL commandExtract of DIR/FULL command $ dir/full sys$system:sysdump.dmp
Directory SYS$SYSROOT:[SYSEXE]
SYSDUMP.DMP;1 File ID: (6365,2,0) Size: 350000/350010 Owner: [1,1] Created: 4-AUG-1999 14:25:12.78 Revised: 29-SEP-2000 12:19:11.22 (66) Expires: <None specified> File organization: Sequential Shelved state: Online Caching attribute: No_caching
ReadaheadReadaheadReadaheadReadahead
XFC WILL by default attempt to readahead where it deems it to be appropriate.
VCC_READAHEAD controls this The SHOW MEMORY/CACHE=VOLUME command gives the
number of readaheads. Enabled/Disabled on a system wide basis, if disk queue lengths are
already high, and if cache misses occur on the readaheads, may want to turn it off.
XFC/VCC are client basedXFC/VCC are client basedXFC/VCC are client basedXFC/VCC are client based
Will NOT cache served I/O – MSCP served or QIOServer Served – on the Server.
Third Party products do exist which are LBN caches. Some time in the future QIOServer may implement
an LBN Cache.
Using SCSI tools in SYS$ETCUsing SCSI tools in SYS$ETCUsing SCSI tools in SYS$ETCUsing SCSI tools in SYS$ETC
Volatility is again an issue. Most modern disk devices have on disk caching
ability. If have recovery mechanisms, then could consider
this. Benchmark above which took 7 hours 11 Mins now
takes just 5 Hours 25 Minutes 04 Secs – though of course there is now a risk component, if the power was to fail.
Local Writeback Cache Enabled on DiskLocal Writeback Cache Enabled on DiskLocal Writeback Cache Enabled on DiskLocal Writeback Cache Enabled on Disk
show mem/cache=vol=iss$
System Memory Resources on 1-OCT-2000 08:26:43.52
Extended File Cache Volume Statistics:
_$9$DKB500: (DISK$ISS_V1), Caching mode is VIOC Compatible Open files 115 Closed files 110 Files ever opened 1311 Files ever deposed 1086 Allocated pages 143548 Locks acquired 1200 Total QIOs 4433091 Read hits 3218148 Virtual reads 3372990 Virtual writes 1056810 Hit rate 72 % Read aheads 6728 Read throughs 3372655 Write throughs 1056810 Read arounds 335 Write arounds 0 Ave Disk I/O Resp Time incl cache hits (microseconds) 2217
Writeback CachingWriteback CachingWriteback CachingWriteback Caching
Not yet implemented Future enhancement for XFC On a per file, directory or Volume basis
– SET FILE/CACHE=NO_CACHING– SET FILE/CACHE=WRITETHROUGH– SET FILE/CACHE=WRITEBACK
For application writers, $FLUSH command – equivalent of SYNCH in Unix file systems.
Ian PercivalIan PercivalIan PercivalIan Percival XFC Project Leader OpenVMS Cluster Engineering [email protected]