Advanced Technical Support, eServer, pSeries

ATS Performance Analysis 6/1/2006 © 2005 IBM Corporation

Virtual Memory Management – Tuning Parameter: lru_poll_intervalAIX 5.3 ML1+ or AIX 5.2 ML4+

Barry J. SaadHarold R. Lee

Advanced Technical Support - pSeries

© 2005 IBM Corporation2 ATS Performance Analysis 6/1/2006

VMM Tunable: lru_poll_intervalPurpose:

Specifies the interval at which the lrud should poll for pending interrupts. Note: Each memory pool has an independent timer.

Range:10 – (default) the lrud runs until the page stealing goal has been achieved. (Note: depending on the maintenance level the default value is 0. Recommend changing to 10, which is the default for later maintenance levels.x = (1 – 60000) – the lrud runs for x msecs before polling for interrupts

When to Tune:Monitoring with vmstat, if you have activity in the ‘sr’ -scan and ‘fr’ free columns, which indicates that lrud is running. Monitoring with vmstat, you notice that the number of processes on the block queue temporarily increase while lrud is running.Monitoring with tprof, you notice that lrud is running.Monitoring with filemon, you notice that you have longer than normal read response times at the physical layer.

lru_poll_interval

lrud starts with a requestfor memory andcalculated goal and runsto completion before re-enabling interrupts.

Set t

he v

alue

ot 0

and

lrud

work

s to

com

plet

ion Set the value to non-zero

Poll for interrupts

lrud starts with a requestfor memory andcalculated goal.

External timer event -lru_poll_interval occurs

Handle the timer event:lrud stops working - pollsand processes interrupts.Complete

1 - Pop Timer

2 -

Poll & Process

3 - Return

Advanced Technical Support - pSeries

© 2005 IBM Corporation3 ATS Performance Analysis 6/1/2006

Virtual Memory Manager Tunable Parameter: lru_poll_interval

The lru_poll_interval parameter was introduced at ML4 of AIX5L 5.2 and ML1 of AIX5L 5.3. The parameter tells the page stealer lrud whether it should stop working and poll for interrupts or continue processing until the current request for free frames has been satisfied.

The process starts when the VMM needs memory and the lrud process starts running to steal memory. During the process the VMM will determine how many pages it needs to steal, including enough pages to get above maxfree. Remember the lrud will run under the following conditions:

Memory Manager Tuning Parameters

maxperm and strict_maxclient

minperm maxclient and strict_maxclient

lru_poll_interval

Kernel Counters numperm numclient

1. The number of frames on the free list drops below minfree.

2. Hard limit on either maxperm and/or maxclient. The hard limit is set with strict_maxperm and strict_maxclient to 1.

3. A Workload manager trigger is hit.

Tuning Options

By setting lru_poll_interval = 0, you’re telling lrud not to poll.

By setting lru_poll_interval =10, you’re telling lrud to poll every 10 msecs (default).

By setting lru_poll_interval = x (1 – 60000), you’re telling the lrud to stop working every x msecs and poll and process any interrupts.

Advanced Technical Support - pSeries

© 2005 IBM Corporation4 ATS Performance Analysis 6/1/2006

When does this value need to be tuned? The reason to change the default value is that you’ve decided and/or determined that the lrud is blocking interrupts, which is having a negative impact on system performance.

So how can we make this determination?

Virtual Memory Manager Tunable Parameter: lru_poll_interval

Monitoring with vmstat, if you have activity in the ‘sr’ - scan and ‘fr’ free columns, which indicates that lrud is running. Now, interrupts are not block during the entire time lrud is running – the daemon starts and stop numerous times as individuals requests are made. Also, keep in mind that your system may have multiple lrud to manage each memory pool.

lrud is a threaded application – use the ps –o THREAD –p <lrud_pid> to determine how many threads are running.

Monitoring with tprof, you notice that lrud is running.

Monitoring with filemon, you notice that you have longer than normal read response times at the physical layer. Particularly true if you can correlate to lrud running

Monitoring with trace, you notice a long time between the physical device start and the physical device iodone. Check the iodone hook, which will include a end timer, if available. If you don’t get an end timer value the corresponding start event was most likely missed.

221 dd 0.076764604 0.001664 SCDISKDD iodone: rhdisk1 bp=F10001195007F400 B_READ B_AGE [13554 usec]

221 dd 0.080413470 0.001818 SCDISKDD iodone: rhdisk1 bp=F10001195007AC00 B_READ B_AGE

When tuning increase and decrease the value in 2-5 msec increment and note the external performance impact.

Finally, as with all tunable parameter changes, you should note the behavior of the system before and after the change- in this case I would use ‘vmstat’. If performance improves leave the parameter as set. If performance degrades return the parameter to the original value.

Remember, it is not uncommon to “pass” the optimal point in the tuning process, when you do - dial back the last set of changes and take a baseline snapshot using the the “vmo –a or vmo -L” command.