Best Practices for Performance Tuning JBoss Enterprise Application Platform 4.3

www.jboss.com

2 In search of hIgh-performance applIcatIons

2 performance tunIng prIncIples

2 Why tune for performance?

3 Performance priorities

3 A word about performance benchmarks

5 JBoss eap 4.3 tunIng

5 Connection pooling

6 Thread pooling

8 Object and component pools

9 Logging

9 Caching

10 JBoss EAP performance tuning summary

11 Linux-specifictuning:Largememorypages

11 Linux-specifictuning:Tuningthevirtualmemorymanager

11 Database tuning update

12 Case study

13 applIcatIon performance tunIng: a contInuous process

13 appendIx: usIng large-page memory (lInux-specIfIc InstructIons)

Best practices for performance tuning JBoss enterprIse applIcatIon platform 4.3 tIps and trIcks for optImIzIng your applIcatIon’s performance

2 www.jboss.com

Best practices for performance tuning JBoss Enterprise Application Platform 4.3

in search of high-performance applications

Overall,performancetuningisaveryimportantpartofcreating,maintaining,anddeployingasuccessfulbusinessapplication.Whetheryouarebuildingcustomapplicationsordeployingcommercial,off-the-shelfsolutions,youwilllikelyneedtotunetheapplication,thedatabase,themiddleware,orallthree.Infact,75%of performance issues originate with the application itself.

Whenorganizationsselectapplicationmiddleware,performanceisalwaysoneoftheirmostimportantselectioncriteria,ifnotthemostimportant.InmanycasesourcustomerstellustheychoseJBossEnterprise Application Platform (EAP) because of its superior performance. They know that many users ofJBossEAPareachievingsuperiorapplicationperformancedayafterday.

Togetthemostfromyourcompany’sinvestmentinmiddleware,developersandarchitectsneedtoknow thespecificwaystheycanachievesuperiorperformancewithJBossEAP.Whilewewouldallliketothink|thatanapplicationcouldperformwellstraightoutofthebox,thisisnotusuallythecase.Applications canhavewidelyvaryingcharacteristics,andwhilesomeapplicationsmightperformwellwithdefaultmiddlewaresettings,otherswillnot.

IfyouarenewtoJBossEAPorperformancetuning,thispaperwillintroduceyoutobestpracticesthatcanhelpyouavoidcommonperformancepitfallsasyouprepareyourapplicationforproduction.Ifyou’reanoldhandatapplicationperformanceissues,youknowthattechnologyisconstantlychanging.Youmaybenefitfrom an update on best practices in JBoss EAP performance tuning.

performance tuning principles

Why tune for performance?

Performance was once considered just another feature of an application. Today it is frequently considered themostimportantcharacteristicoftheapplication—onethatcanhaveasignificantimpactonyourbusinessandyourproductivity.Consideryourreactiontoaslowwebsite.Ifyou’relikemostpeople,youbecomefrustrated,losepatience,andgoelsewhere.Ifthatcompanyiscountingonrevenuefromwebsales,ithasnotonlylostyourattention—ithaslostbusiness.Evenforinternalapplications,poorperformancecanaffectproductivityifusershavetowaitordealwithunpredictablebehavior.Atbest,theymightfeelannoyed,losealittletime,orformanegativeopinionoftheirITdepartments.Atworst,businesstransactionsmaybelost,orcustomersmaygowithoutimportantserviceifusersmustworkaround a poorly performing application.

But user experience is not the only reason to tune for performance. A well-performing application will generallyusefewerhardwareandsoftwareresources.Acompanycanoptimizeitsinvestmentinhardwarewhenapplicationsaretunedappropriately,whetherthatmeansusingoldersystemslonger,purchasingnewsystemsthataremoremodestlysized,orusingfewersystemsoverall.Onthesoftwareside,awell-performingapplicationwillgenerallyneedtousefewerCPUcountsorsoftwarelicenses,nomatterwhattypeofsoftwareisinvolved.Reducingsoftwarecostscansavethecompanysignificantmoneyovertime.

www.jboss.com 3


performance prIorItIes

Superiorperformancecomesfrommanylayersoftheapplicationstack,notjusttheapplicationserver.Inmanycases,thewaytheapplicationisdesignedandhowitconnectstothedatabaseandothersoftwarecomponentscanhavealargeimpactonoverallapplicationperformance.Manyorganizationsspendmoretimetuningtheircustom-builtapplicationsanddatabasesthantheirunderlyingapplicationservers.Sokeepinmindthatasuperiorlyperformingapplicationservermayhaveonlyaminorimpactontheoverallperformance of your applications.

a Word aBout performance Benchmarks

Some organizations rely on industry-standard performance benchmarks when selecting middleware. While benchmarkscanhelpvendors,theycandeceiveyouasanevaluatorbecausebenchmarkapplicationsareusuallyverydifferentfromtheapplicationsyouwillruninproduction.Thecurrentbenchmarkforapplicationservers,forexample,SPECjAppServer2004,isa2004applicationthatdoesn’ttakeadvantageofmanyrecentdevelopmentsinJava.Butasidefromthat,everyapplicationisdifferentandthesystemsonwhichbenchmarkapplicationsarerunmaybeverydifferentfromyours.Understandinghowagivenbenchmarkrunsmaytellyouverylittleabouthowyourapplicationwillrunonyourhardwarewithyoursettings.Beforeyourelyonabenchmark,youneedtoinvestigatethesoftwareandhardwareconfigurationsused,theserverandnetworkconfigurations,thesettingsused,thearchitectureoftheapplication,andhowallofthesecomparewithyourownenvironment.Werecommendthatbeforeselectingasystem,youtestitwithanapplicationandhardwareconfigurationasclosetoyoursaspossible.1

performance tuning principle #1: understand your performance requirements

Thefirststepintuningyourapplicationforperformanceistounderstandtheconditionsunderwhichitwillneedtoperform.Ifyourapplicationisareplacementforanexistingsolution,thenyourorganizationalreadyhassignificantexperiencewiththatsolution.Chancesareyouhavemetricsavailablesuchasthenumberofusers,thenumberoftransactionsperday,thevariationsintransactionloadortypeoverthecourseofaday,week,month,oryear,andsoon.

Ifyouaredeployingacompletelynewsolution,youwillneedtostudythatapplication’sbusinesscontextverycarefully.Themoreyouunderstandexactlyhowyourapplicationwillbeused,themoresuccessfulyourperformancetuningwillbe.Sometimesassumptionsdonotreflectreality.Inonecase,adevelopmentteamcreatedandtesteditsapplicationbasedonanassumedworkloadof60,000transactionsperday.Thefirstdayinproduction,6milliontransactionsoccurred.

performance tuning principle #2: plan for peaks, not averages

Asyouexamineperformancerequirements,oneofyourgoalsshouldbetodevelopaprofileofyourapplication’sworkloadwithspecialattentiontothepeaks.Forexample,manybusinessapplicationsexperiencedailypeaksinthemorningandafternoonwithavalleyduringthemiddleofthedaywhen

1 Foramorein-depthexplanationoftheprosandconsofperformancebenchmarks,seeDoPerformance Benchmarks&ComparisonsMatter?—AGuidetoAssessingApplicationServerPerformanceResults.

4 www.jboss.com


employeesareeatinglunch.Someapplicationsexperiencepeaksattheendofthemonth,quarter,or season.Yourapplication’sworkloadprofilewilldependonthespecificsofyourbusiness,butyoushouldalwayspayparticularattentiontoperiodsofpeakworkload.Oneofthebiggestmistakesdevelopersmake istorelyonanaverage(averagedailyworkload,forexample).Averagesarenotsufficienttoensurethatyour application will perform during periods of peak load.

performance tuning principle #3: always instrument your application

Allapplicationsshouldbeinstrumentedtoprovideinformationforperformanceanalysis.Businessconditions,includingcustomerbehaviorandworkloadcurves,canchangedramaticallyovertime,soevenifanapplicationonceperformedwell,itmaynotperformwellundertoday’sconditions.Ifyourunintotrouble,andyourapplicationhasnotbeeninstrumentedforperformance,thenyouhavenoeasywaytoknowwhereyourproblemsare.Butifyourapplicationisappropriatelyinstrumented,thenyou’llbeabletomonitorchanges in business conditions easily and tune your applications to match before problems occur.

Inthepast,inordertoinstrumentanapplication,adeveloperneededtoembedcodewithintheapplicationitself.Todaymanysolutionsprovideinformationwithoutrequiringdeveloperstocode.WithJBossEAP,thecallstatisticsinthecontainerprovideyouthenumberofcalls,concurrentcalls,mincalltime,maxcalltime,averagecalltime,andthestandarddeviationofcalltimes.Hibernatestatisticsprovidequeryexecutiontimes,andJBossONMonitoringcandisplayallofthisinformationandgraphtheresultsinrealtime.Additionally,numerousthird-partyapplicationperformancemanagement(APM)productsareavailable fromcertifiedRedHatpartnerstoprovideapplicationperformanceinformationaswell.

Finally,forperformance-criticalsituationswhereoneoftheothertoolsdoesn’tprovidetheinformationyouneed,youcanwriteyourowninstrumentationusingtheJBossAOPframework.Thisframeworkoffersquitea few features that enable you to see exactly what is happening at runtime. Whether you choose to write yourowninstrumentationwilldependontheapplication,yourskillsandavailabletime,andtheimportanceoftheapplication’sperformancecharacteristicstoitsoverallsuccess.

performance tuning principle #4: understand where your application spends its time

One important reason to instrument your application is to understand where time is being spent. While you wanttoknowabouttimeacrosseachlayerofyourapplicationstack,thetoolswithinJBossEAPwillhelp youunderstandonlypartofthisequation.Ifyourapplicationisspendingtoomuchtimeinthedatabase, forexample,thenyoumayneedtofocusonyourdatabasestatisticstopindowntheproblem.

Byunderstandingwhereyourapplicationspendsitstime,youwillbeabletoavoidthe“shotgunmethod”of performance tuning — trying multiple solutions to common problems without knowing whether any of themarerelevanttoyourproblem.Youmightsolveyourproblemthisway,buttheprobabilityislow.Manydeveloperswhoassumetheyknowwhatiscausingtheirperformanceproblemaremistaken,resultinginproblemsthatpersistformonthsorsometimesyears.Thatiswhyhavinganobjectivewaytoknowwhereyourapplicationisspendingtimewillhelpyouavoidspendingtoomuchtimesolvingperformanceproblems.

www.jboss.com 5


performance tuning principle #5 replicate or model your production environment

Asyouworktounderstandexactlyhowyourapplicationwillperforminproduction,youwillneedtorunitinatestenvironment.Somecompaniesmakeitstandardpracticetoimplementatestenvironmentthatisanexactreplicaoftheirproductionenvironment.Thisisanidealapproachinmanyrespects,becausedevelopersdon’thavetowonderhowtheirapplicationswillscaleforlargersystems.

Formanyorganizations,though,thecostofreplicatingtheproductionenvironmentisasignificantobstacle.Theirtestenvironmentstypicallyemploysmallersystems.Ifthisisyoursituation,thenyouwillneedtocreateamodelthatdefinestherelationshipbetweenperformanceinyourtestenvironmentandperformanceintheproductionenvironment.Thatis,youneedtomodelhowtheapplicationwillscale.Asyoudoso,keepthefollowinginmind:

do not assume a linear relationship.• Performance doesn’t often scale in a linearly. Vendors may tellyoutheyexperiencelinearresults,butforrealproductionapplications,thisisrarelythecase.

Be conservative in creating your model, and use actual historical data wherever possible.• Ifyoucanbaseyourmodelondatafrompastexperiencewithotherapplications,youwillbeabletorefineitovertimeandincreaseyourconfidenceinitsaccuracy.

JBoss eap 4.3 tuning

While75%ofallperformanceproblemsaretheresultoftheapplication,notthemiddlewareorthe operatingsystem,youstillneedtoknowhowtotunesettingsinthemiddlewaretoimproveperformance andthroughput.Dependingonyourapplication,oneormoreofthefollowingmayneedsomeattention:

Connection pooling•

Thread pooling•

Object and component pools•

Logging•

Caching•

Thefollowingsectionsprovideanoverviewofeachoftheseareas.

connectIon poolIng

Connection pooling and thread pooling are the most important areas to consider when you want to maximize throughputonmodernhardware.Intermsofsystemresources,databaseconnectionsareexpensivetosetupandteardown.Butinspiteofthis,someapplicationscreateanewconnectiontothedatabasewitheveryqueryortransactionandthenclosethatconnectionimmediately.Thispracticeaddsagreatdealofoverheadtotransactionprocessingandcanleadtopoorperformance.

6 www.jboss.com


TotakeadvantageoftherobustconnectionpoolinginJBossEnterpriseApplicationPlatform,startbyadjustingconnectionpoolsettingsonthedatasourcedefinitionsthatyoucansetupinthedeploydirectory.Settheminimumpoolsizetothelevelyouwanttotunefor,thensetthemaximumatleast25-30%higher.Don’tbeconcernedaboutsettingthemaximumtoohigh,asthepoolwillshrinkautomaticallyyoudon’tneedthat many connections.

Todeterminethepropersizing,youcanmonitoryourconnectionusage.Toosmallapoolwillalsothrottletheapplication,astheEAPwillqueuetherequestforadefaultof30,000milliseconds(or30seconds)beforegivingupandthrowinganexception.Ifyoustartseeingalotof30-secondtimeouts,thatisastrongcluethatyouneedtolookatyourconnectionpooling.YoucanmonitortheconnectionpoolutilizationfromtheEAPJMXconsole,fromJBossON,orfromdatabase-specifictools.

Hereisanexampleofconnectionpoolsettingsforadatasource:

<datasources> <local-tx-datasource> <jndi-name>MySQLDS</jndi-name> <connection-url>jdbc:mysql://<host>:3306/Schema</connection-url> <driver-class>com.mysql.jdbc.Driver</driver-class> <user-name>someuser</user-name> <password>somepassword</password> <exception-sorter-class-name>org.jboss.resource.adapter.jdbc.vendor.MySQLExceptionSorter</exception-sorter-class-name> <min-pool-size>75</min-pool-size> <max-pool-size>100/max-pool-size> ...  <metadata> <type-mapping>mySQL</type-mapping> </metadata> </local-tx-datasource> </datasources>

thread poolIng

Thread pooling is the next most important area to consider as you tune your application for performance. JBossEAPhasrobustthreadpooling,butbeforeyoucansizethethreadpoolsappropriately,youneedto know how they are used and which ones might be affecting your application’s performance. The characteristicsofyourspecificapplicationwilldeterminewhichthreadpoolsareusedandwhichones mightbecomebottlenecks.Thiscanvarysignificantlyfromapplicationtoapplication.Thetablebelowprovidesasummaryofhoweachthreadpoolisused.

www.jboss.com 7


removing connectors

Ifyouarecertainthataparticularconnectorwillnotbeusedinyourapplication,werecommendremoving it.Forexample,manyapplicationsuseeithertheHTTPdthreadpoolorthemod_jkthreadpoolbutnotboth.Soyoucanremovetheoneyoudon’tneed.

monitoring and tuning thread pools

MonitoryourthreadpoolsviatheEAP4.3JMXConsole,whichdisplaysnotonlythenumberofactivethreads for each pool but also the queue size. We recommend that you adjust thread pool settings through JBossOperationsNetwork,asitallowsyoutodefinesettingsthatpersist.SettingsadjustedviatheJMXconsolewillnotsurviveareboot.IfyouusetheJMXconsole,pleaseremembertogobackandeditthefile if you want the adjustments to be permanent.

thread pool Where Is It defIned? hoW Is It used?

System thread pool Injboss-service.xmlintheconfdirectory ForJNDInaming—thedefaultsetting isfineformostcases

HTTPDthreadpool in JBoss Web

Intheserver.xmlfileunder<server>/deploy/jboss-web.deployer

WhenmakingHTTPrequestsdirectly to EAP

AJP thread pool Intheconnectorsectionofserver.xml WhenmakingHTTPrequeststhroughmod_jk

JCA thread pool (also called the Work Managerthreadpool)

<server>/deploy/jbossjca- service.xml

InconjunctionwithJMS,asJBossMessagingusesJCAinflowastheintegration into EAP

JBossMessagingthreadpool (for remote clients)

<server>deploy/jboss-messaging.sar/remoting-bisocket-service.xml

Pools the TCP sockets

JBossMessagingthreadpool(inJVMclients)

Notdirectlyconfigurable Pool is bounded by the number of MDBsyouhavedefined

EJB3(sameJVM) ClientsinthesameJVMwillrun onwhateverthreadpooltheyare already using

Forexample,awebrequestcomes in through the AJP connector. When itcallsanEJB3bean,itwillcontinueexecuting on the AJP connector thread pool.

EJB (remote clients) <server>/ejb3/deployer/META-INF/ jboss-service.xml

8 www.jboss.com


example: http thread pool

<Connector port=”8080” address=”${jboss.bind.address}” maxThreads=”250” maxHttpHeaderSize=”8192” emptySessionPath=”true” protocol=”HTTP/1.1” enableLookups=”false” redirectPort=”8443” acceptCount=”100” connectionTimeout=”20000” disableUploadTimeout=”true” />

example: mod_jk or aJp thread pool

 <Connector port=”8009” address=”${jboss.bind.address}” protocol=”AJP/1.3” emptySessionPath=”true” enableLookups=”false” redirectPort=”8443” maxThreads=”200” />

example: Jca thread pool

<mbean code=”org.jboss.util.threadpool.BasicThreadPool” name=”jboss.jca:service=WorkManagerThreadPool”>  <attribute name=”Name”>WorkManager</attribute>  <attribute name=”MaximumQueueSize”>1024</attribute>  <attribute name=”MaximumPoolSize”>100</attribute>  <attribute name=”KeepAliveTime”>60000</attribute> </mbean> <mbean code=”org.jboss.resource.work.JBossWorkManager” name=”jboss.jca:service=WorkManager”> <depends optional-attribute-name=”ThreadPoolName”>jboss.jca:service= WorkManagerThreadPool</depends> <depends optional-attribute-name=”XATerminatorName”>jboss:service= TransactionManager</depends> </mbean>

oBJect and component pools

Object pools and component pools are essentially the same thing. Their settings represent the number of objectinstances.ForEJB3twotypesofpoolsaredefinedin<server>/deploy/ejb3-interceptors-aop.xml.ThesearetheThreadLocalPoolandtheStrictMaxPool.Bydefault,StatelessSessionandStatefulSessionBeansusetheThreadLocalPool,whichisbackedbyanInfinitePoolwithnomaximumsize.Therefore,itgrowsaccordingtovolumeinyourapplication.Thishasthedistinctadvantageofnotneedingtobetuned.Bydefault,MessageDrivenBeans(MDBs)usetheStrictMaxPool.Thispoolactuallyobeysamaximum,will

www.jboss.com 9


queueuprequestswhenthatmaximumhasbeenreached,andwilltimeoutanythinginthequeueifthereisnotanavailablereferencefromthepool.Inthiscase,thesystemwillthrowanexceptionandiftheproblemoccurredinmid-transaction,youwillexperienceatransactionrollback.Giventheimpactfailedtransactionscanhaveonyourbusiness,werecommendthatyoumonitortheStrictMaxPoolcloselyviatheJMXconsole.

loggIng

Developersshouldtakefulladvantageoflogginginthedevelopmentandtestingphasesoftheapplicationlifecycle.Inproduction,however,loggingcancausebottlenecks.Youwanttobesurethatloggingprovidesyou with useful information without hurting application throughput. Consider making the following changes asyoupromoteyourapplicationintoproduction:

turn off console logging in production.• InEAP’sdefaultconfiguration,consoleloggingisenabled,whichmeansyouhavetheopportunitytoseeallthelogsfromyourIDE.Inproduction,thisisanexpensiveprocesswithunbufferedI/O.Whilesomeapplicationsmaybefinewithconsolelogging,high-volumeapplicationsbenefitfromturningitoff.JBossEAP4.3offersanewconfigurationset,namedProduction,whichgivesdevelopersabetterstartingpointforcreatingaproductionenvironment.IntheProductionconfiguration,consoleloggingisturnedoff.

turn down logging verbosity.• Thelessyoulog,thelessI/Owilloccur,andthebetteryouroverallapplicationthroughputwillbe.Loggingisalwaysatradeoff,sothinkcarefullyabouthowmuchloggingyou really need in production.

use asynchronous logging.• This can make a big difference for high-throughput applications. With asynchronouslogging,logmessageswillgointoaqueueandcontrolreturnstotheapplicationasif the logging had been completed. Then a separate thread executes the log operations from the queue.

Wrap debug log statements with• If(debugEnabled()). This simple practice can make a huge differenceifyourapplicationcontainsalotofdebuglogstatements.Withoutthisconditionset,yourapplicationcreatesallofthestringobjectsforeachofthelogstatements,andLog4jcreatestheLoggingEventobjectforeachlogstatementregardlessoftheloglevelthatissetbecausetheloglevelischeckedonlyafteralloftheseobjectshavebeencreated.InsomecasesthiscanleadtocreationofthousandsandthousandsoftemporaryStringandLoggingEventobjects,resultinginmemoryandgarbage collection issues and reducing throughput dramatically. By placing a conditional wrapper aroundyourdebuglogstatements,youcanensurethatunnecessarylogprocessingdoesnotaffect your throughput.

cachIng

Caching,whileoftenveryhelpful,maybeoneofthetrickiestareastotunecorrectly.JBossCacheisanintegralpartoftheEAPinfrastructure,andyourapplicationcanuseittocacheanythingyoulike.Cachingisespeciallyvaluableforapplicationsthatperformalotofreadoperationsagainstdatathatiseithercompletelystaticordoesn’tchangefrequently,suchasreferencedata.Forapplicationswithheavyuseofwriteoperations,cachingmaysimplyaddoverheadwithoutprovidinganyrealbenefit.Ifnotconfiguredproperly,Cachingmayactuallyreducethroughput.

10www.jboss.com


One of the easiest potential performance enhancements you can make is to cache EJB 3 entities. To definewhichentitiesyouwantcached,modifythefilepersistence.xml that you deploy with your EJB 3application(anexampleisshownbelow).Youdefinethecachesizeandevictionpolicyinthefileejb3-entity-cache-service.xmlfoundinthedeploydirectory.Thesedefinitionsmayrequiresometrial anderrortogetright.Rememberthatyouareworkingwithlimitedheapspace,andamisguidedcachingstrategy can be worse than none at all.

Evictionpolicywilldependonthespecificsofyourapplication.Youcandefinethecacheastransactional orread-write.Withverylargedatasets,notethatcachingmaynotprovidenoticeableperformancebenefits.Inthiscase,shrinkingthecachescanimproveperformance.Also,ifyourapplicationisverywrite-heavy,itmaynotbenefitfromcaching.Testingvariouscachingandnon-cachingconfigurationswillhelpyoudetermine this.

example: settings for caching in persistence.xml

<persistence> <persistence-unit name=”services” transaction-type=”JTA”> <provider>org.hibernate.ejb.HibernatePersistence</provider> <jta-data-source>java:/MySQLDS</jta-data-source> <properties> <property name=”hibernate.default_catalog” value=”EJB3”/> ... <property name=”hibernate.cache.provider_class” value=”org.jboss.ejb3.entity.TreeCacheProviderHook”/> <property name=”hibernate.treecache.mbean.object_name” value=”jboss.cache:service=EJB3EntityTreeCache”/> <property name=”hibernate.ejb.classcache.services.entities.Customer” value=”read-only”/> <property name=”hibernate.ejb.classcache.services.entities.Inventory” value=”transactional”/> ... </properties> </persistence-unit> </persistence>

JBoss eap performance tunIng summary

Tosummarize,keepthefollowingperformancerecommendationsinmindwhentuningJBossEAP4.3.

DefinedatasourcesinthedeploydirectoryandtakeadvantageofEAP’srobustconnectionpooling.•

Understandwhichthreadpoolsareactuallyusedbyyourapplication,removepoolsthatarenot •needed,monitorthenumberofactivethreadsandthequeuesize,andadjustpoolsizeifneeded.

Ifyouareusingmessage-drivenbeans,monitortheStrictMaxPoolcloselytoensurethatthemaximum•object pool size is not a bottleneck.

www.jboss.com 11


Employdifferentloggingstrategiesfordevelopmentandproduction.Especiallyforhigh-volume•applications,turnconsoleloggingoff,reduceloggingverbosity,useasynchronouslogging,andalwayswrap debug log statements with If (debugEnabled ()).

Takeadvantageofcachingifyourapplicationisread-heavyormakessignificantuseofreferencedata.•Avoidcachingforverylargedatasetsorwrite-heavyapplications.

lInux-specIfIc tunIng: large memory pages2

Fortoday’s64-bitsystems,useoflargememorypagescanimproveperformancesignificantly.Thedefaultmemorypagesizeistypically4KB.Whenyouareaddressinglargeamountsofmemory,thisquicklyadds uptoalargenumberofmemorypages—justonegigabyterequires262,144memorypages.That’salotfor asystemtokeeptrackof,whichtranslatestoalotofsystemoverhead.

Asidefromhelpingyouavoidtheoverheadofmappingsomanymemorypages,largememorypagesonLinuxcannotbeswappedtodisk.Thisisamajoradvantagebecausehavingyourheapspaceswaptodiskcanwreckhavocontheperformanceofyourapplication.

Largepagesupportbeginsat2MBandcanrunashighas256MBonsomehardwarearchitectures.Thesenumberswillvary,andyouwillneedtofindoutthevaluesforyourspecificserver.AllthemajorJVMsystemssupportlargememorypagesonLinux.Becauseitcanbetrickytosetup,weprovidesomespecificinstructions in the appendix.

Whenyouuselarge-pagememory,keepinmindthatthememoryisnotavailabletoapplicationsingeneral.Tootherapplications,yoursystemwilllookasthoughithashadmemoryremovedfromit,becausethismemorywillbededicatedtoyourspecificapplication.Refertotheappendixformorespecificinformationaboutconfiguringyourapplicationtouselargememorypages.

lInux-specIfIc tunIng: tunIng the vIrtual memory manager

InLinux,thevirtualmemorymanageristunable.Insomecases,youcanachieveperformancebenefitsbychangingthesettings.Youwon’tbeusingthefilesystemcachemuch,andyoudon’twantthesystemtofavorthefilesystemcacheoveryourapplications,whichcansometimesoccurwiththedefaultsettings.

In/etc/sysctl.conf you can set vm.swappinessto1topreventapplicationsfrombeingswappedto disk when there is memory pressure.

dataBase tunIng update

Moderndatabases,especially64-bit,areextremelyefficientatcachingdata.Intheearlydaysof64-bitdatabases,largebuffersizeswouldslowperformanceduetoelongatedsearchtimes,butthisisnolongertrue.Inourtesting,wehaveexperiencedOLTPapplicationswithverylargebuffercachesthatshowverygood results.

Consideryourapplication’scharacteristics—especiallyitsread/writeratios—whendecidingwhetherandhowtousedatabasecaching.Themajorityofapplicationsweseetodayareread-intensive,asmostapplicationsdrivetheirbusinesslogicbyreadingdatafromthedatabase.Themoreread-intensivetheapplicationis,themorecachingcanhelp.Iftheapplicationiswrite-heavy,orthedatasetisverylarge(fromadatawarehouse,forexample),thenalargecachewon’thelpandmayslowdowntheapplication.

2 WhilethissectionprovidesLinux-specificsettings,theprinciplesoflargememorypageuseareapplicable to any 64-bit system.

12 www.jboss.com


YoushoulduseDirectI/Oifyourdatabasesupportsit.Especiallywithalargecacheorawrite-intensiveworkload,youshouldavoiddoublebufferingwiththefilesystembuffercache.NotethatMySQL5documentationindicatesthatqueriesmayslowdownbyuptoafactorof3whenusingDIRECTI/O,butwehavenotexperiencedthisincaseswhereaproperlysizedbufferpoolisused.IfyouareusingDIRECTI/O,youshouldbesuretousethevirtualmemorysettingmentionedearlier.WealsosuggestusingasynchronousI/Oifyourdatabasesupportsit.

case study

TodemonstratethedramaticresultsthatcanbeachievedwithJBossEAPperformancetuning,weperformed an experiment using a sample application.

WebeganwithJBossEAP’sdefaultconfiguration(withoneminorexception),alongwithallLinuxparametersattheirdefaults.UsingGrinder,anopensourceJavaload-testingframework,wemeasuredthehighestthroughputwecouldachievewithallofthesesettings.Usingthesameapplication,wethenappliedmanyoftheoptimizationsdiscussedhere,bothtoJBossEAPandtotheoperatingsystem(inthiscase,Linux),andmeasured throughput again.

TheapplicationwasanEJB3applicationwithtwoservletsfortheUI,statelessandstatefulsessionbeans formostofthebusinesslogic,amessagedrivenPOJOforsomeasynchronousprocessing,andentitiesforthepersistence.Alltestswereperformedwith3.5GBheapandadatasourcewithsufficientconnections in the pool.

Thefollowinggraphfirstillustratesasinglevirtualuserasabaselineformeasuringscalabilityasweaddedusers.Asonewouldexpect,thefirsttwobarsarestatisticallyequalregardlessoftheperformanceoptimizationsmade.Thenexttwobarsshowhowmanyvirtualuserswecouldachievebeforewewerenolongerat85%oflinearscalability,asmeasuredbyatransactionspersecond(TPS)meanfromthebaseline(thebarsontheleft).Youcanseethatintheun-optimizedcase,wewereabletoreach25virtualusers, andintheoptimizedcase,wewereabletoreach75virtualusers.Weachievedathreefoldincreaseinthenumberofvirtualusersthatthesamesystemcouldsupport,simplybyoptimizingsystemperformance.

80

70

60

50

40

30

20

10

0

Baseline EJB 3 application 1 virtual userTPS (mean)

No optimizations

Baseline EJB 3 application 1 virtual userTPS (mean)Optimized

Top throughput test 25 virtual users

TPS (mean)No optimizations

Top throughput test 75 virtual users

TPS (mean)Optimized

JBoss eap performance tunIng

www.jboss.com 13


application performance tuning: a continuous process

Tuninganapplicationforoptimalperformancecanensureapositiveuserexperience,promotebusinessproductivity,andhelptooptimizeuseofhardwareandsoftwareresources.Performancetuningisnota one-timetask,butanongoingprocessthatensuresawell-performingapplicationasbusinessconditions andsystemtechnologychangeovertime.Applicationdevelopersandarchitectsshouldalwaysbepreparedtotunetheirapplicationsforperformancebothbeforeandaftertheyareputintoproduction.Asalways, themorebusiness-criticalanapplicationisandthehigherthevolumeoftransactionsitmustsupport,themore important performance tuning will be to your business.

Whentunedinaccordancewiththecharacteristicsofyourapplication,JBossEnterpriseApplicationPlatform(EAP)canprovidesuperiorapplicationperformance.Thispaperhasgivenyouanoverviewof basic performance principles as well as an introduction to performance tuning best practices for JBoss EAP.KeepinmindthatthesetechniquesapplytoanyoftheRedHatplatformsthatJBossEAPsupports:JBossEnterprisePortalPlatform,JBossEnterpriseSOAPlatform,andJBossEnterpriseBRMS.

ForadditionalinformationonJBossEAPperformancetuning,pleasereferto:

Managingapplicationperformance:JBossOperationsNetwork•

JBossEAP4.3ServerConfigurationGuide•

JBossEAPOptimizationandTuningAssistanceServices•

JBoss EAP 4.3 Product Documentation•

appendix: using large-page memory (linux-specific instructions)

Thisappendixcontainsaprocedureandexampledesignedtohelpyoutakeadvantageoflarge-pagememoryforyourapplications.Seethesection“Linux-specifictuning:Largememorypages”earlierinthispaperforanoverview.

1. tell the Jvm to use large-page memory.

SunJVMandOpenJDKrequirethefollowingoption,passedonthecommandline,touselargepages:

-XX:+UseLargePages

TheSuninstructionsleaveitatthat,butifyoudonothingelseyouwillmostlikelygetthefollowingerror:Failed to reserve shared memory (error-no=12)

The following sections describe additional steps you should complete.

14 www.jboss.com


2. set kernel parameters.

Set three kernel parameters in /etc/sysctl.confasfollows:

kernel.shmmax = n • Where n is equal to the number of bytes of the maximum shared memory segment allowed on the system.YoushouldsetittoatleastthesizeofthelargestheapsizeyouwanttousefortheJVM.Alternatively,youcansetittothetotalamountofmemoryinthesystem,andyouwillneverhave torevisitit.

vm.nr_hugepages = n• Wherenisequaltothenumberoflargepages.Youwillneedtolookupthelargepagesizein /proc/meminfo.

vm.huge_tlb_shm_group = gid• WheregidistheIDofasharedgroupIDfortheusersyouwanttohaveaccesstothelargepages. This setting enables you to limit access to the large memory segment.

3. set limits.conf parameters.

Next,setthesememlocklimitsin/etc/security/limits.conf:

<username> soft memlock n <username> hard memlock n

where <username>istheruntimeuseroftheJVM,andnisthenumberofpagesfromvm.nr_hugepages multiplied by the page size in KB from /proc/meminfo.

4. persist your settings.

Enterthecommand:

sysctl –p

Thisensuresthatthesettingsyoucreatedinstep3willsurviveareboot.

5. reboot.

WhentheOSallocatesthesepages,itmustfindcontiguousmemoryforthemortheoperationwillfail. Arebootwillpreventthisproblem.

6.Confirmpageallocation.

Whenlargepagesareallocated,/proc/meminfo will display a non-zero number for HugePages_Total. Ifyoudonotseeanon-zeronumber,thenyouarenotusingthelargepages,andsomethingisconfiguredincorrectly.

www.jboss.com 15


WehavesometimesseenaproblemwithMYSQLwheretheSELinuxpolicywaspreventingitfromaccessingthe large pages. Check /var/log/messages for avc_denied messages (error-no=13 “permission denied”) in the mysqld.log.

example: setting large memory pages

Consideraserverwith8GBofmemory.Wewillallocate6GBtobesharedbytheJBossEAPJVMand aMySQLdatabase.

Thepagesizeis2MB,asshownin/proc/meminfo (Hugepagesize: 2048KB)

Configuration:/etc/sysctl.conf

Changemaximumsharedmemorysegmentsizeto8GB. •kernel.shmmax = 8589934592

Add the gid to the • hugetlb_shm_grouptogiveaccesstotheusers. vm.hugetlb_shm_group = 501

Add6GBin2MBpagestobesharedbetweentheJVMandMySQL. •vm.nr_hugepages = 3072

Calculations:

1024*1024*1024*8=8589934592

(1024*1024*1024*6)/(1024*1024*2)or6GB/2MB=3072pages

Configuration:/etc/security/limits.conf

AddthelimitsformemlocktoallowtheJVMandMySQLtoaccessthelarge-pagememory.•

jboss soft memlock 6291456 jboss hard memlock 6291456 mysql soft memlock 6291456 mysql hard memlock 6291456 root soft memlock 6291456 root hard memlock 6291456

Calculations:

3072largepages*2048KBpagesize–3072*2048=6291456

Configuration:/etc/group

AddJBossandMySQLuserstothe501(• hugetlb) group in /etc/grouptogiveuserspermission to attach to the shared memory segment.

JBoss sales and InquIrIes NORTHAMERICA

1–888–REDHAT1 www.jboss.com

Copyright©2009RedHat,Inc.RedHat,RedHatEnterpriseLinux,theShadowmanlogo,JBoss,MetaMatrix,and RHCEaretrademarksofRedHat,Inc.,registeredintheU.S.andothercountries.Linux® is the registered trademark ofLinusTorvaldsintheU.S.andothercountries.

www.jboss.com # 1388350_1109

Best Practices for Performance Tuning JBoss Enterprise Application Platform 4.3

Documents

Transcript of Best Practices for Performance Tuning JBoss Enterprise Application Platform 4.3