AIX Virtual Processor Folding is Misunderstood (AIXpert Blog)

Sign in | RegisterIBM

Technical topicsTechnical topics Evaluation softwareEvaluation software CommunityCommunity EventsEvents Search developerWorks

Blogs SearchThis Blog

AIXpert BlogAIXpert Blog

0LikeLike

Tweet 1

AIX Virtual Processor Folding is Misunderstoodnagger | Aug 8 2011 | Comments (4) | Visits (35209)

This mysterious AIX CPU Folding area is often misunderstood, so below is what I knowfrom osmosis from talking to various guru level developers over the last 10 years. SharedProcessor virtual machines (LPARs for the old fashioned) have a setting called VirtualProcessors (or VP for short). This is the number of physical CPUs that the virtualmachines can spread out across - in fact, I prefer to call it the "spreading factor" as it ismuch more obvious what it means. This can be the upper threshold for the number ofCPUs that can be used by your virtual machine:

For Capped virtual machines, the Entitlement is the maximum of compute cycles that can be used.For Uncapped virtual machines and if there are "spare" CPU cycles in the Shared CPU Pool (or othervirtual machines are not requiring their full entitlement) then the VP number is the limit of the total numberCPU cycles that can be used.

But AIX does not automatically spread out over all the available virtual processors, if it does not have too, as thatis not efficient. If an Uncapped virtual machine has, for example, Entitlement of 8 CPUs and Virtual Processorcount of 10 CPUs but only needs at the moment 2.5 CPUs to easily provide CPU time to all the runningprocesses/program then it "folds" away the unneeded 7 virtual processors and runs on just 3.

Why fold? I am not a AIX kernel developer (well not now anyway) nor a spokes person for development teambut there is not a lot of information about folding. I think this is because it is a AIX kernel internal optimisation andso there is no need to make it public nor document it. I first noticed folding taking place while developing nmon.On the Power 5, 6 and 7 machines, we could clearly see that AIX would stop scheduling work to some virtualprocessors when they were not needed and thought "these developers are very clever" and was impressed.There are multiple reasons to fold:

If you schedule processes to a lower number of physical CPUs (cores) then the memory is cached better(often called the caches are hotter) in the Level 1, 2 and 3 memory cache levels = hotter caches meansmore progress as data is loaded in to CPU registers a lot faster.

1.

The unused physical CPUs can carry on running other virtual machine workloads without interruption =making them more efficient.

2.

No longer scheduling the virtual machine to a CPU it does not need just to perform pointless contextswitches to the virtual machine to then give up the CPU milliseconds later = more work done.

3.

Detecting folding: We also noted that these folded virtual processors do actually get some CPU cycles. nmonused the libperfstat AIX library to extract the performance numbers and we found that instead of billions of clockticks on the processor these folded ones have a few - in the 50 to 100 ticks sort of range. We think this is a tinyamount of house keeping going on like collecting the processor hardware stats, perhaps a clock interrupt and Ithink that if a processor starts a device driver to action an adapter to perform an operation that the later interruptis returned to the same processor (which might now be folded). This is a little guess work on my part so don'tquote me - please! Back to nmon: there is no official AIX folding statistics from any interface that I have comeacross. nmon deduces the folding number by monitoring the physical clock ticks and heuristically (I like thatwords) determines if it is folded or not. In C programming terms, are the number of ticks below a threshold weworked out empirically (another good word) by sitting there and watching the numbers while tweaking thenumbers of processes running.

Now for the complicated bit - SMT! When a virtual processor is folded all the Simultaneous MultiThreading(SMT) threads are switched off together - the different threads are always running in the same virtual machine.This made the heuristics in the nmon code tricky as it has to check all the threads on the same CPU are doingpractically nothing. This was written when Power6 was the current processor and SMT=2 was the maximum. Ofcourse, when Power7 came along with SMT=4 and the code had to be reworked - in fact there is a nmon releasethat runs on Power7 where the code had not been updated and gets the folding count hopelessly wrong but itsfairly obvious when you see it (you can't fold away more CPUs than you actually have!) - install a service pack tofix this.

What does AIX actually do when it folds? We have seen that the virtual processors are still there and seen byAIX and occasionally running a few clock cycles. So they are not stopped or released. AIX is simply no longerscheduling processes to run on them. This gives the hypervisor a hint that at the moment it does not have toschedule the virtual machine on them. The AIX kernel to Hypervisor interface and mechanisms are definitely asecret, so I have no clues at this level.

About this blogAIXpert Blog is about the AIX operating systemfrom IBM running on POWER based machinescalled Power Systems and software related toit like IBM Systems Director, PowerVM forvirtualisation and PowerSC for security plusperformance monitoring and nmon

Related posts

Renaming jfslogs and...Updated July 10 0 0

Investigating High C...Updated June 27 2 2

Want ot get sub_ms r...Updated June 25 0 0

Installing multiple ...Updated June 18 0 0

PureApplication Syst...Updated June 13 0 0

Links

YouTube vidoes on AIX, POWER7,...

Tags

Find a Tag

- & 5.2 6 6.1 6.3 770 780 795 active aem

affinity aix aix6 aix7 ame analyser cpu

director entitlement ethernet firmware

hmc hypervisor linux lpar memorymonitoring nmon on performance pools

power power6 power7 powervmprocessor ram server shared storage

systems tools topas upgrade vios virtualvm workload wpar

Cloud List

Recent tweets

Follow @mr_nmon

4

ShareShare

AIX Virtual Processor Folding is Misunderstood (AIXpert Blog) https://www.ibm.com/developerworks/community/blogs/aixpert/entry/...

1 of 3 7/13/2015 8:42 PM

Comments (4) Add a Comment More Actions

When to fold? AIX does this slowly but seems to monitor the CPU use for a period of time (from myobservations a few seconds which is a long long time in CPU terms) and determines that the CPU use is steadyor dropping and its is safe and efficient to fold a CPU off. It then waits to see what happens next and may thenfold away a further CPU. It is maths that larger virtual machines with many 100's or 1000's of active processesand threads of execution do not have sudden jumps of CPU requirements but the workloads sort of "flows in andout" and CPU use is smoothed out.

When to unfold? AIX again monitors (it is probably the same algorithm) the CPU use. When it notices aconsistent high use of the current unfolded CPUs it decides that unfolding could help the throughput of theprocesses and unfolds a CPU. If there is a sudden peak in runnable processes it does not immediately unfold.This is because it can already deal successfully with short term transitory peaks as normal via the run queues. Iffact, over aggressive unfolding could slow the applications down. When a CPU is unfolded its caches will beempty then a happy process on the running CPUs with hot cache is moved to the new unfolded one with coldcaches - it will spend the next few milliseconds loading the cache with program, data, stack and heap memorylines before it can get back to full speed. If we then find the temporary peak has passed, AIX will fold the CPUagain - it was all rather a waste of time and the process moved twice actually got slower. This is why it holds off alittle time before unfolding and makes sure it is a genuine growth in the demand for CPU time. It gives me theimpression that, for slowly growing workloads, AIX unfolds at something like a virtual processors once a secondbut that is an observation rather than a fact.

Why are we monitoring Folding? Well, I was keen to have this in nmon because it gives us good clues aboutwhether we have the right Entitlement for our virtual machine. If (particularly monitoring long term like over amonth), we always find that we have Folded virtual processors and particularly during our peaks in workload thenperhaps we have the Entitlement set too high. It could be dropped to let other workloads be added to the samemachine. On the other hand, if we have a critical production virtual machines that has for long peak periodsFolding = zero then perhaps we should consider raising the Entitlement to make sure this virtual machine alwayshas sufficient CPU cycles.

Folding is Leaking out! This advanced optimisation technique inside AIX was an internal secret at one time buthas become known. I suspect my nmon monitoring might have accelerates that a little :-) But there are a fewcomments in the manual now and there are a few Folding tuning parameters available.

The advanced AIX scheduling tuning command, "schedo" has these options:

AIX6+ vpm_fold_policy AIX6+ vpm_xvcpus AIX6 TL06+ & AIX 7 also has vpm_fold_threshold (in the schedo command but not in thedocumentation yet). This is the utilisation threshold at which AIX begins to unfold. In the past this was80% and in the latest AIX release been lowered.

I would not go fiddling with these unless you have a particular problem to fix and have recommendationsfro AIX support. You can easily make performance worse with non-default settings. Check the currentsettings and that try are the default with (as the root user): schedo -L

In the manual pages:

Virtual Processor discussion in the manuals

mpstat -s 1 9999 is an alternative way to "see" Folded virtual CPUs.

Read the AIX online manual for more information on these.I hope this helps, thanks, Nigel Griffiths.

Tags: power7 powervm power6 hypervisor aix folding aix6 aix7 virtual processor

Add a Comment More Actions

Comment PermalinkRWolter commented Apr 13

Well Done. Congrats for this article

Comment Permalinkabdurixit commented Feb 18 2013

Excellent article, thanks Nigel.

Comment PermalinkRagaza commented July 24 2012

Can u pls help me to understand "cold cache" Thanks


2 of 3 7/13/2015 8:42 PM

About

Help

Contact us

Submit content

Feeds

Newsletters

Report abuse

Terms of use

Third party notice

IBM privacy

IBM accessibility

Faculty

Students

Business Partners

Select a language:

English

中文

日本語

Русский

Português (Brasil)

Español

Việt

Previous Entry Main Next Entry

Comment Permalinkcggibbo commented Aug 9 2011

Excellent post. One of the better explanations of processor folding. Thanks Nigel.


3 of 3 7/13/2015 8:42 PM

AIX Virtual Processor Folding is Misunderstood (AIXpert Blog)

Documents

Transcript of AIX Virtual Processor Folding is Misunderstood (AIXpert Blog)