HPEReferenceArchitectureforMicrosoft ExchangeServer2016onHPE ProLiant BL460cGen9andD6000StorageDeploy 7,500 mailboxes with Exchange Server 2016on RAID-less JBOD storage

Technical whitepaper

Technical whitepaper

ContentsExecutivesummary 3

Solution overview 4

Designprinciples 5

High availability (HA) and disaster recovery (DR) 5

Scaling thesolutionbeyond 7,500 mailboxes 6

Solution components 6

Enclosuresand I/Omodules 6

Servers 6

Operating systemandapplication software 8

Best practicesandconfiguration guidancefor thesolution 8

Exchange2013PerformanceHealth Checker Script 12

Capacity andsizing 13

Performance 13

Capacity 13

Analysisandrecommendations 13

Summary 15

Implementingaproof-of-concept 15

AppendixA: Bill of materials 16

Resourcesandadditional links 17

Technical whitepaper Page 3

Executivesummary

Thiswhite paper is intended for customerswho want tomigratetoMicrosoft®ExchangeServer 2016andwho havestandardized onHPE

BladeSystem servers.ExchangeServer2016 isoneof the latest releasesin along lineof ExchangeServer releases.Over theyears,Exchangehas

changed inanumber of ways. It moved froma32bit toa64bit architectureandhaschangedstorageinput/output (I/O) requirementssuch that

slower,lower cost diskscan beutilized. Thehigh availability model wasmodifiedsuch that Exchangenativedataprotectioncanbethedefault high

availability solution. Manyother architectural advanceshavebeenimplementedthat changedsystemrequirements,someof which changethe

focusfromvery highperformancestorage,tohigher performanceserverswith RAMand CPUcapabilities toaddressthenewapplication

requirements.

Deploying ExchangeServer 2016onolder serverscan requiremoreserverstobedeployed,dependingon theuserprofileand othersolution

parameters.Inoneexample,using earlier generationsof serverscould result inneeding fromtwo tosevenmoreserverstosupport thesame

ExchangeServer 2016workloadoperating on5HPEProLiant BL460cGen9servers.Inaddition toserver costs,thiswould alsoincrease

administration, networking andother infrastructurecosts.Deploying ExchangeServer 2016on themost recent HPEProLiant generationcan help

reducethenumberof serversrequired,while alsoreducing networking,power and administrative costs.An in-placeupgradeusing existing

hardwareused tohost earlier versionsof ExchangeServer isnot possible,sonewserver hardwareandstoragecapacity is required todeploy

ExchangeServer 2016.

Thiswhite paperdescribesdeploying Microsoft ExchangeServer 2016onHPEProLiant BL460cGen9serverblades in anHPEBladeSystem

c7000 enclosurewith direct attachedstorage(DAS),utilizing HPED6000 diskenclosures.This implementationensuresahighly available

solutionwhile minimizing costs.This testedsolutionsupportsacustomer usecasewith 7,500 10GBmailboxesat aprofile of 150 total messages

sent and received per dayper mailbox.Rather thanusing RAIDtoprovidehighavailability (HA) at thestoragelayer, thissolution utilizes the

nativedataprotection featuresof ExchangeServer 2016 through implementationof adatabaseavailability group (DAG) with threecopiesof each

databaseamongfiveservers.Thedatabasesand transaction logsarestoredon individual RAID-lessdisksin a“just abunch of disks”,or JBOD

configuration.

Deploying thesolution onHPEBladeSystemcapitalizeson thepower andcooling savingsof theBladeSysteminfrastructurealong with

simplifying datacenteroperationsand offeringgreaterdensity, sharedresources,scalability and greateroverall efficiency.For moreinformation

about thebenefitsof deploying HPEBladeSystem,pleaserefer to the IDCTop 5Reasons toMovetoBladeSystem,presentation:

http://h20195.www2.hpe.com/v2/GetDocument.aspx?docname=4AA5-7828ENW

TheMicrosoft PreferredArchitecture(PA) for Exchangeisutilized asaguidein deploying this solution.Wherepossible thePAis followed.

Encryption of dataat rest utilizing BitLocker isalsodiscussed,along with keysystemcomponents,suchasTrustedPlatformModule(TPM) 1.2,

andCPUsthat support the Intel®AES-NI instruction set toaccelerateencryption andreducethestorageperformanceimpact of encryption.

Insummarizing testing with Microsoft ExchangeJetstressandLoadGen,thiswhitepaper showsthat thissolutioncansupport the target 7,500

mailboxeswith a150 messagesperdayper mailboxprofile in several different scenarios.Thesolutionsupportsatwoserver outagescenario,

whereevenwith two serversunavailable, thesolution cansupport apeak load of 300 messagespermailbox perday.

ExchangeServer 2016 wasreleasedin September of 2015.It advancesonExchangeServer 2013with moreefficient cataloging andsearch

capabilities,simplifiedserverarchitectureanddeployment models,fasterand morereliableeDiscovery, and expandeddatalossprevention

features.Sizing for Exchange2016 isvery similar to thesizing processfor Exchange2013,which isdemonstratedwith thetest results later in this

document,ascompared to thereferencearchitectureat:http://h20195.www2.hpe.com/V2/GetDocument.aspx?docname=4AA6-2902ENW

Technical whitepaper Page 4

Solutionoverview

Thissolution isbuilt around an HPEBladeSystem c7000 enclosure,aD6000 diskenclosure,fiveBL460cGen9servers,and internal componentsto

bediscussedlater in thisdocument.Thissolutionalsoutilizes thenativedataprotection featuresof Exchange2016with multiple copiesof each

database.Storagecostsarereducedby using direct attachedstorage(DAS) in ajust abunchof disks(JBOD) configuration for Exchange

databasesand logs,which meansthat noRAIDprotection isprovidedby thestoragesubsystem.Eachphysical 6TBSAS7.2K3.5indiskhoststhree

copiesof variousdatabases,and eachdatabasehasthreecopiesspreadamong thefiveservers.

Figure 1 . Logica l Exchange Server 2016 di agram with DAG, servers and some database copies.

In theabovefigure,eachExchangeserver ishosted onaProLiant BL460cGen9server in aBladeSystemc7000 enclosure,which isattached toa

D6000 diskenclosuretoprovidestoragecapacity for theExchangedatabases.

In thissolution, eachserver hasapair of boot/systemdiskswhich areprotected in aRAID1pair,andeachserver isprovisioned fourteendisks

fromtheD6000,which areutilized in aJBODconfigurationwhereeachphysical disk is itsown arrayand logical drive.With twelve disksfor

databases,theextratwodisksperserverarefor recoveryspaceand repair in theevent of adisk failure.

Ageneral viewof thesolution in asingleBladeSystemc7000 enclosureand D6000 disk enclosureis outlined belowin Figure2.Thedetailsof

thesolutioncomponentsarediscussed in thenext section.

Technical whitepaper Page 5

Figure 2 . Front and rear views o f B ladeSystem c7000 enclosure wi th f i v e server blades and D6000 disk enclosure

Each of the fiveBL460cGen9serverscontain96GBof RAM, two E5-2630 v3CPUs,oneHPEFlexFabric20Gb 2-port 650FLBFlexibleLOM,one

Smart ArrayP244br controller andoneSmart ArrayP741mcontroller.TheSmart ArrayP244br controller isused for the internal boot/system

drives in eachserver,while theP741mcontroller is installed in mezzanine slot 2inorder to interfacewith HPE6Gb SASSwitches in I/Obays5and

6 of thec7000 enclosure.For Ethernet connectivity thec7000 enclosurecontainsHPEVirtual Connect FlexFabric10Gb/24-port Modules in I/O

bays1and 2.

Designprinciples

Microsoft ExchangeServer isoneof the,if not the,most widely usedbusinessproductivity applications.It isamissioncritical servicefor many

businesseswherebusinessquickly comestoahalt if theservice isnot available. In order tomeetbusinessrequirements,theExchangeservice

should bedesignedaround ServiceLevelAgreements(SLA) which can includeperformancelevels,uptimerequirements,capacity requirements,

and recovery time.

Highavailability (HA) anddisaster recovery (DR)Thissolutionutilizes thenativedataprotection featuresof Exchangeasone layer of highavailability (HA).In thissolution, up to twoserverscan

fail or otherwisebeofflineandeachdatabasewill still haveat least oneactivecopy toserveusers.Each server usesaRAID1logical drive for

WindowsBoot/Systemand for Exchangetransport databases,but theExchangemailboxdatabasesand logsareheld onsinglephysical disksona

RAID0 logical drive.Theserversaresized toprovide theExchangeserviceevenwhen two serversareunavailable.Theload increaseson thethree

remaining servers,but they aresized tohandlethat increasedload in the failurescenario.

Another aspect of highavailability ismanagingsinglepointsof failure.TheHPEBladeSystemc7000 enclosurein thissolution hasredundant

power supplies,fans,OnboardAdministrator (OA) modules,andredundant Virtual Connect FlexFabric10Gb/24-port andSASmodules.Each of

Technical whitepaper Page 6

theseredundant componentsneedstobemanagedaccordingly, suchasthepower suppliesbeingconfiguredin asuitableHAmodeandbeing

suppliedby separateand redundant power sources,and theOAmodulesbeing onseparatenetworks.

The storage subsystem also hassome level of HA built-in. The D6000 disk enclosure has two drawers,each with 35 LFFdisk bays.Each drawer

has redundant power supplies, redundant fans,and redundant data paths to the c7000 enclosure.The 6TBSAS7.2K3.5in disk drives each have

two I/Oports,soalossof asingleSASpath will not causelossof accessto thediskdrives.

TheMicrosoft PArecommendsnot using NICteaming in order tosimplify the failover model.WhilethePAisMicrosoft’srecommendationsbased

ontheir deployment methods,not all installations will beat thesamescaleasthat deployment.Insmaller scale,anExchangedeployment may not

spandatacenters,or evenrackswithin thesamedatacenter,soother deployment optionsshouldbeconsidered.

For network connectivity redundancy of each of the BL460c Gen9 servers,while NICteaming isnot recommended in the Microsoft PA, the most

efficient way to implement network HA in this configuration is to useWindows NICteaming.With NICteaming, the failure of any single NICon a

server will not impact thesolution andneither will the failureof either of theVirtual Connect FlexFabric10Gb/24-port Modules.

For further HA,thesolution canbedeployed acrossmultiplegeographicsites.Becauseof thedetailsof suchadeployment,such asnamespace

design,activation scenarios,quorumplacement,etc.thistype of deployment isoutside thescopeof thisdocument.

Scaling thesolutionbeyond7,500mailboxesThesolutioncanbescaledbeyond 7,500 mailboxessimplybyusing thissolutionasa7,500 mailboxbuilding block,meaningthat for each7,500

mailbox increment,another fiveserversand D6000 aredeployed.Asthesolutiongrows,multiple c7000 enclosureswould beutilized and

databasecopiesshouldbedistributed amongthec7000 enclosuressuch that not all copiesof adatabasewould resideonasingleD6000 disk

enclosureor c7000 enclosure.Sincethissolutionwasnot testedbeyond 7,500 mailboxes,further discussionof this isoutsidethescopeof this

document.

Solutioncomponents

ThisExchange2016solution for 7,500 10GBmailboxesutilizes thefollowing HPEBladeSystemcomponents:c7000 enclosure,Virtual Connect

FlexFabric10Gb/24-port Modulesand BL460cGen9.TheD6000 diskenclosureisoutsideof thec7000 enclosure.Each of thesearecoveredin

moredetail asfollows.

Enclosures and I/OmodulesThissolution isdesignedaround theHPEBladeSystem c7000 enclosurewith six platinumpower suppliesandtwo Onboard Administrator (OA)

modules.For network I/O,twoVirtual Connect FlexFabric10Gb/24-port Modulesareutilized for redundant Ethernet connectivity within the

enclosure.While eachserver utilizesoneNICasper theMicrosoft PreferredArchitecture,connectingall of theNICstoasinglenon-redundant

switch or I/Omodulewould createasinglepoint of failurefor all of theservers.In thissolution, two Ethernet I/Omodulesareusedtoprovide

networkHA within theenclosureand toexternal top of rackswitches.

For connectivity to the D6000 disk enclosure HPE6Gb SASSwitch Dual Pack for HPEBladeSystem c-Class is installed in I/Obays 5and 6 of the

c7000 enclosure.These I/Omodulesare connected to the D6000 asshown in Figure 2above.Four of the HPEExt Mini SAS1m Cables are used

tocabletheD6000 to theSASI/Omodules.

ServersHPEProLiant BL460cGen9server bladesareutilized in this solution.Eachof theserversis configuredwith 96GBof RAM, andwith two Intel

Xeon®E5-2630 v3CPUs,for 16corestotal.Asnoted in theExchangeTeamBlog (http://blogs.technet.com/b/exchange/archive/2013/05/06/ask-

the-perf-guy-sizing-exchange-2013-deployments.aspx) sizing guidance,theseserversaredeployed with Hyper-Threading turned off.Also, as

noted in thesesizing recommendations:https://technet.microsoft.com/en-us/library/dn879075(v=exchg.150).aspx,theserversshould alsobe

configuredin theBIOStoallowtheOStomanagepower,andWindows®should beset to thehighperformancepower plan.

Eachserver isconfiguredwith apair of internal 2TBSAS7.2K2.5inHDDsfor boot andsystem.Thedisksin theD6000 for Exchangedatabases,

logs,recoveryand automaticreseedingarediscussedin theStoragesectionbelow.

An internal viewof theBL460c Gen9server isshown in Figure3.More informationabout theBL460cGen9serversisavailableat:

hpe.com/servers/bl460c

Technical whitepaper Page 7

Figure 3 . In ternal view o f BL460c Gen9

Storage

The D6000 disk enclosure is utilized for database and transaction log files,and recovery,maintenance and automatic reseeding capacity.The

D6000 can hold a total of seventy large form factor (LFF) 3.5 inch drives.Those seventy drives are divided between two drawers of thirty-five

driveseach.Thissolution usesall seventy drive bays in theD6000 with seventy HPE6TB6GSAS7.2KrpmLFF(3.5-inch) drives.

Toutilize thestoragein theD6000,eachof theExchangeserversisconfiguredwith aSmart ArrayP741mcontroller.TheRAIDfeaturesof the

Smart Arraycontroller arenot usedon thesedatabaseand log disksasthey areconfigured asJBOD,whereeachphysical disk is itsown array

andRAID0 logical drive.Eachserverusestwelve disksfor active and passivedatabasecopies,with another two disksper server for recovery,

auto-reseedor maintenancespace.

TheD6000 isshownbelowin Figure4.Moreinformationabout theD6000 isavailableat:

http://www8.hp.com/us/en/products/disk-enclosures/product-detail.html?oid=7390970

Informationabout deployment andcabling of theD6000 isavailableat:

http://h20565.www2.hpe.com/portal/site/hpsc/template.PAGE/public/psi/manualsResults/?sp4ts.oid=5307027and

http://h20564.www2.hpe.com/portal/site/hpsc/public/kb/docDisplay/?docId=c01956983

TheMicrosoft PreferredArchitectureoutlinesusing WindowsBitLocker for at rest data protection.For effectiveuseof BitLocker,eachserver

should beconfiguredwith theTrustedPlatformModule(TPM) 1.2.Thiseasestheuseof BitLocker by storing and securing theencryption keys

local to theserver without requiring aBitLocker password each timetheserver boots.Toalsoeasetheperformanceimpact of BitLocker,the

CPUsused in thissolution includethe Intel AES-NI instruction set,which isusedby BitLocker to reduceCPUandperformanceimpact.More

informationabout Intel AES-NI isavailableat:intel.com/content/dam/doc/white-paper/enterprise-security-aes-ni-white-paper.pdf

Thissolution wastestedwith unencryptedstorageand with storageencryptedbyWindowsBitLocker and theperformancedifferencewas

negligible.Informationondeploying BitLocker is available at:https://technet.microsoft.com/en-us/library/hh831713.aspxand

https://technet.microsoft.com/en-us/library/jj612864.aspx

Technical whitepaper Page 8

Figure 4 . D6000 f ront external view.

Operating systemandapplicationsoftwareThis solution is built on WindowsServer 2012R2with all applicableupdatesinstalled viaWindowsUpdateasof thedateof testing (December

2015):Version 6.3,Build 9600.ExchangeServer 2016(build 15.01.0225.042) wasused for thissolution. HPEServicePack for ProLiant version

2015.06.0 was installed oneach of theBL460c Gen9serversfor driver and firmwareupdates.

Microsoft ExchangeLoadGenand Jetstresswereusedasthetest tools for thissolution. LoadGenwasversion15.00.0847.030 andJetstresswas

15.00.0995.000. For Jetstresstherequired.dll and other support fileswereusedfromExchangeServer2016(build 15.01.0225.042).

Bestpracticesandconfigurationguidance for thesolution

Microsoft recommendsdisabling Hyper-Threading whendeploying Exchangeonphysical servers.Thisisnot necessary in avirtualized

environment aslong astheadditional CPUsarenot used in capacityplanning.In thissolution, eachserver hastheBIOSconfigured todisable

Hyper-Threading andtoset thepower profileset to OScontrol with theEnergy/PerformanceBIASset to “MaximumPerformance”.Figures5

through7belowshowthosesettings.

Figure 5 . Disab l ing Hyperthreading i n the server BIOS.

Figure 6 . Set t ing Power Pro f i le to “Custom” and Power Regula to r t o “OS Contr o l Mode”.

Technical whitepaper Page 9

Figure 7 . Set t ing Energy/Pe r formance Bias to “Maximum Per f ormance” .

Key pointTheseExchangeserversshould beconfiguredwith Hyperthreading turned off andwith thepower profileset toOScontrolled.Misconfiguringthesesettingscanhaveanegativeimpact onperformance.

TheBladeSystemSASswitchesmust beconfiguredtoprovision 14 drives for each of thefiveExchangeServers.TheHPE6GVirtual SASmanager

application is launched fromtheHPEBladeSystemOnboardAdministrator web console.ZoneGroupsarecreated,populatedwith drive bays,and

thenassigned toserver bays.Azonegroup iscreated for eachExchangeserverand ispopulatedwith drivebays according to thefollowing table.

Table 1. SAS Zone group con f iguration parameter s .

TheVirtual SASManager providesasummaryviewof theSAStopology asshownbelowin Figure8.

ZONE NAME ASSIGNED DRIVE BAYS ASSIGNED SERVER BAY

Exch01 Drawer 1: Bays 1-14 Server bay 6

Exch02 Drawer 1: Bays 15-28 Server bay 7

Exch03 Drawer 1: Bays 29-35Drawer 2: Bays 29-35

Server bay 8

Exch04 Drawer 2: Bays 1-14 Server bay 14

Exch04 Drawer 2: Bays 15-28 Server bay 15

Technical whitepaper Page 10

Figure 8 . Configured SAS Topology

After thezonegroupsareconfiguredand assignedtoaserverbay, eachserverwill see14 newdiskdriveson theSmart Array P741mcontroller.

In thissolutioneachoneof thosediskdrives isconfiguredasaseparatearray, with asingleRAID0 logical drive.Theview fromoneserver is

shownbelowin Figure9 for arraysAthroughC.ThisconfigurationcontinuesthrougharrayN.

Figure 9 . HPE Smar t Storage Administrat o r configura t ion .

Technical whitepaper Page 11

Theaboveconfigurationcanbeaccomplishedthrough theGUIshown above,or it canbedoneat thecommandline.Thedefault path for the

commandlineutility is C:\ProgramFiles\HP\hpssacli\bin. Oncein that directory,acommandline suchasbelow,canbeutilized toconfigureall 14

driveson theSmart ArrayP741massingledriveRAID0 logical drives:

1..14 | foreach ($_) {$drivestr ing= "51:1:$_" ; wr ite- host " . \ hpssacli.exe c t r l slot=2 create type=ld

drives="$dr ivestr ing" raid=0 s t r ips ize=256"}

OncethedrivesareconfiguredandexposedtoWindowsasnewdiskdrives,theymust beinitialized, mountedtomount points,and formatted.

This solutionutilized themount point of C:\ExchangeVolumeswith eachdiskmountedtoafolder under that asshown in Figure10 below.

Figure 10. Mount point root , mount points , vo lumes and mult ip l e databases per volume

Either theWindowsGUI or PowerShell canbeused to initialize,mount and format thevolumes.Thedetailsof thesePowerShell commandsare

outsidethescopeof thisdocument.Extremecaremust betakenwhen automatingdiskprovisioning in order toprotect disksalready in usethat

maycontainoperatingsystemor other critical information.Onedifferencebetweenan Exchange2016 deployment andanExchange2013

deployment isthat theWindowsvolumesshould be formattedwith theResilient Filesystem,or ReFs,for Exchange2016.

Key pointsOn theSmart Arraycontroller, thelogical drivesshould haveastrip sizeof at least 256KB.When formatting thedisksin Windows,theallocationunit sizeshould beset to64KB.TheSmart Arraycontroller cachewasleft at thedefault of 10%readand90%write cache.

Thissolutionutilizes60 Exchangedatabaseswith threecopiesof thosedatabasesamongthe fiveservers.Figure11showsthedistribution of

thosedatabasesamongtheservers.Thenumbersin thecellsrepresent thepreferred,secondaryand tertiary server designatedfor each

database.Oncethestorageisconfigured,mount pointscreatedanddatabaseand log folderscreatedwithin thosepaths,then thedatabase

distribution among theserversshould beconfiguredasshownbelow.

Technical whitepaper Page 12

Figure 11. Database copy di s t r ibu t io n among th e servers.

Exchange2013PerformanceHealthChecker ScriptMicrosoft providesanExchange2013PerformanceHealth Checker Script that checkscommonconfiguration settingssuchasproduct versions,

pagefilesettings,power plansettings,NICsettings,andprocessor/memory information.Exampleoutput for theExchangeserversisshown belowin

Figure12.It isrecommendedtorun theExchange2013Health CheckerScript periodically toensurethat your environment isoperating at peak

health and that configurationsettingshavenot been inadvertently changed.Microsoft providestheExchange2013PerformanceHealth Checker

Script at https://gallery.technet.microsoft.com/Exchange-2013-Performance-23bcca58.While this script wasinitially releasedfor Exchange2013,it

still providesvaluable information for Exchange2016.

Figure 12. Example Heal th Checker scr i p t output .

Technical whitepaper Page 13

Capacity and sizing

PerformanceTwotoolsareused toplan theperformanceandcapacity requirementsfor thissolution: theHPESizer for Microsoft ExchangeServer 2013

(hpe.com/solutions/microsoft/exchange2013/sizer) and theMicrosoft ExchangeServer RoleRequirement Calculator v7.8,which includes

calculations for Exchange2016(https://gallery.technet.microsoft.com/office/Exchange-2013-Server-Role-f8a61780).TheExchange2010

Processor Query tool (https://gallery.technet.microsoft.com/office/Exchange-Processor-Query-b06748a5) isalsousedwith theMicrosoft

calculator to lookup theSPECInt valuesfor prospectiveCPUs.TheHPESizer for Microsoft ExchangeServer2013wasusedassizing for

Exchange2016 isnot that different thansizing for Exchange2013.

In using either the HPESizer or the Microsoft Calculator, some characteristics of the workload are utilized for performance sizing. The number of

messagessent and received per day per mailbox is a primary workload characteristic.This solution issized around a profile of 150 messagessent

and received per mailbox per day. Using that profile results in the five serverseach sized with two Intel Xeon E5-2630 v3 processorswith 96GBof

RAM.

Table2showstheprofile,CPUandRAMcombinationandwhat theexpectedCPUutilization is for different server failurescenarios.Actual test

resultsarecovered later in thispaper.

Table 2. CPU and RAM con f iguration and expected CPU u t i l i z a t ion.

CapacityThissolution isdesignedsuch that eachof the7,500 mailboxescanbeup to10GBin capacity.With threeactive/passivedatabasecopies,this is

efficiently doneutilizing fourteen6TBLFF7.2KHDDsfor eachof the fiveservers.Thiscapacity isoptimizedby using aRAID-lessJBOD

configurationwherethereisnoRAIDoverheadaswith RAID10 or RAID5logical drives.WhileRAID5minimizesthecapacity overheadof RAID

protection, performanceisnot typically adequatewith RAID5and 7.2KRPMdiskdrives,and with lessmean timebetween failure(MTBF),there is

greater riskof datalosswith 7.2KRPMdrives in aRAID5arrayduring anarray rebuild operation.

Analysis andrecommendationsJetstress test results

Multiple testswereperformedwith Microsoft ExchangeJetstressto test variousaspectsof thestoragesubsystem,asoutlinedbelow.In these

tests,thetarget IOPSper mailbox is 0.13,or 195IOPS/serverwith fiveserversonline,or 325IOPS/serverwith threeserversonline.

• JetstressTest 1–Normal Load,all fiveserversonline.Goal = Target IOPSunder latency thresholds.

• JetstressTest 2–Normal load,threeserversonline.Goal = Target IOPSunder latency thresholds.

• JetstressTest 3–Very high load.Goal = Demonstratenear upper IOPSlimit per server.

MESSAGE PROFILE CPU(2X PER SERVER)

RAM(PER SERVER)

%CPU WITH FIVE SERVERS ONLINE

%CPU WITH FOUR SERVERS ONLINE

%CPU WITH THREE SERVERS ONLINE

150 msg/day/mbx E5-2630 v3 96GB 48% 54% 64%

Technical whitepaper Page 14

Theresultsof thesethreetests areshown in the tablebelow.

Table 3. Summary o f te s t r esul t s f o r Jetstress tes ts .

Table3showsthat thestoragesubsystemeasily meetstherequiredIOPSfor the target profileandhasmuch I/Otosparebeforereaching 17.6ms

databaseread latency.Withahard limit of 20ms,there isevenmoreheadroomtospare.

TheseJetstresstestshaveshown that thestoragesubsystemcansatisfy thestorageI/Oneedsof this solution and that there isI/Oheadroomin

thesolution.

Thefollowing Microsoft ExchangeLoadGentest resultswill addresstheCPUand memoryrequirementsandperformanceof thissolution.

LoadGentest results

Microsoft ExchangeLoadGenwasusedtomorefully test thisExchangeServer 2016solution. WhileJetstresssimulatesExchangeServerstorage

I/Oonaserver that isnot running Exchange,LoadGensimulatesclient loadonserversactually runningExchange2016.Thistype of testing allows

analysisof CPUandRAMutilization, latencyandresponseof variousExchangesubsystems,andmeasurement of actual messagessent and

receivedby each mailbox toensurethetarget profile isbeing simulatedasaccurately aspossible.

Theloadof 150 messages/day representstheaverageload per mailbox,but peak timescan frequently exceedthat average,soagenerally

acceptedpracticeis to test at twice theaveragelevel tosimulatepeakusage.Testswere thusrun at aneffectiverateof 300 messages/dayto

simulatethispeak impact andareshownbelow

For thissolution, LoadGenwasusedtosimulatethreescenarios.

• LoadGenTest 1–Normal load,all fiveserversonline.Goal = meet target profileandanalyze CPU,RAMandExchangesubsystems.

• LoadGenTest 2–High stressload, fiveserversonline.Goal = meet target profileand analyze CPU,RAMand Exchangesubsystems.

• LoadGenTest 3–High stressload, threeserversonline.Goal = meet target profileand analyze CPU,RAMandExchangesubsystems.

JETSTRESS TEST 1 JETSTRESS TEST 2 JETSTRESS TEST 3

I/O Profile Normal – 150 msg/day Normal – 150 msg/day Very high

Target IOPS / Server 195 325 Near Maximum

Number of ServersOnline

5 3 5

Achieved IOPS / Server 327 454 1247

Database Read IOPS / Server 226 314 861

Database Write IOPS / Server 101 140 386

Average Database Read Latency

(ms)

5.80 6.6 17.6

Average Database Write Latency

(ms)

0.17 0.29 1.2

Transaction Log Writes / Sec 2.1 1.7 8.4

Transaction Log Write Latency (ms) 0.07 0.08 0.14

Technical whitepaper Page 15

Table 4. Summary o f te s t r esul t s f o r LoadGen t ests 1 – 3.

In thepeak load scenarioswhere theCPUisslightly higher thanthedesired thresholdof 70%,adeeper look at theCPUutilization showsthat

content indexing isutilizing asignificant amount of CPU.In thescenarioswith lower CPUutilization, thenoderunner processes,which arethe

content indexingprocesses,wereabout 18%of the total CPUutilization. Inscenarioswith higher CPUutilization, thenoderunner processes

representedabout 34%of theCPUutilization.

TheLoadGentesting showed that thesolutioncansupport the target numberof mailboxesat thetarget profile,even in peak load scenarios,and in

failurescenarioswhere two serversareofflinefor maintenanceor becauseof anunplanned outage.

Summary

Properly sizing an ExchangeServer 2016solution canbechallenging,andsupport becomesmorechallenging if it isnot sized properly.Thiswhite

paper outlined anExchangesolution designedandsized for 7,500 mailboxesof up to10GBeach.Testingshowed that thestorage,CPU, RAMand

networking subsystemscansupport thissolutionat the150 messagesper dayper mailboxprofile for which it wasdesigned in normaloperations

with all fiveserversonlineand in scenarioswhereup to twoserversareoffline.Thesolutionwasalsodesignedand tested for peakload scenarios

of 300 messagesperdayper mailbox in order tohandlepeak loadsat thebeginning of thework day,or during other high loadtimes.

With the architectural changes of Exchange Server over the last several generations,Exchange requires more CPU and RAM resources.This

need can best be met by the latest CPU and architecture of the HPEProLiant BL460c Gen9 servers.Based on CPUcomparisons in multiple

sizing scenarios,deploying with the latest generation of ProLiant servers can reduce the number of servers required by two to seven servers

dependingon thespecificsof theworkload.

Implementing aproof-of-conceptAs a matter of best practice for all deployments,HPErecommends implementing a proof-of-concept using a test environment that matches as

closely as possible the planned production environment. In this way, appropriate performance and scalability characterizations can be obtained.

For help with aproof-of-concept,contact anHPEServicesrepresentative(hpe.com/us/en/services/consulting.html) or your HPEpartner.

LOADGEN TEST 1 LOADGEN TEST 2 LOADGEN TEST 3

I/O Profile Normal – 150 msg/day Peak – 300 msg/day Peak – 300 msg/day

Measured messages/day 145 295 307

Number of serversonline

5 5 3

Average CPU utilization % 53 73 79

MS Exchange RPC Client Access Connection

Count

2944 2943 5058

MS Exchange RPC Client Access Active User

Count

1070 1358 2311

MS Exchange RPC Client Access RPC

Operations/sec

468 879 1490

MS Exchange RPC Client Access RPC Averaged

Latency

4.6 6.5 6.1

Network Interface MBytes sent/sec 3.0 5.7 7.8

Network Interface MBytes received/sec 2..81 5.5 5.0

Average database read latency (ms) 8.2 10.6 9.5

Average database write latency (ms) 0.32 0.25 0.38

Technical whitepaper Page 16

AppendixA:Billof materials

NotePart numbersare at time of testing and subject to change.The bill of materials doesnot include complete support options or other rack andpower requirements. If you have questions regarding ordering,please consult with your HPEReseller or HPESales Representative for moredetails.hpe.com/us/en/services/consulting.html

Table 5. B i l l o f materi a l s

QTY PART NUMBER DESCRIPTION

1 681844-B21 HPE BLc7000 Platinum CTO with ROHS Trial IC License Single Phase

6 733459-B21 HPE 2650W Plat Ht Plg Pwr Supply Kit

2 571956-B21 HPE Virtual Connect FlexFabric 10Gb/24-port Module for c-Class

BladeSystem

1 456204-B21 HPE BLc7000 Onboard Administrator with KVM Option

1 BK764A HPE 6Gb SAS Switch Dual Pack for HPE BladeSystem c-Class

4 407337-B21 HPE Ext Mini SAS 1m Cable

1 517520-B21 HPE BLc 6X Active Cool 200 FIO Fan Opt

5 488069-B21 HPE TPM Module Kit

5 727021-B21 HPE BL460c Gen9 10Gb/20Gb FLB CTO Blade

5 700764-B21 HPE FlexFabric 20Gb 2-port 650FLB FIO Adapter

5 726782-B21 HPE Smart Array P741m/4GB FBWC 12Gb 4-ports Ext Mezzanine SAS

Controller

5 726994-L21 HPE BL460c Gen9 E5-2630v3 FIO Kit

5 726994-B21 HPE BL460c Gen9 E5-2630v3 Kit

30 726719-B21 HPE 16GB 2Rx4 PC4-2133P-R Kit

5 761871-B21 HPE Smart Array P244br/1G FIO Controller

10 765466-B21 2TB 12G SAS 7.2K 2.5in 512e SC HDD

1 K2Q12A HPE D6000 w/70 6TB 6G SAS 7.2K LFF (3.5in) Dual Port MDL HDD 420TB

Bundle

Technical whitepaper Page 17

Signup forupdates

Ratethisdocument

© Copyright 2016 Hewlett Packard EnterpriseDevelopment LP.Theinformation contained herein issubject to change without notice. The

only warranties for HPEproducts and servicesareset forth in theexpress warranty statements accompanying such products and services.

Nothing herein should beconstrued asconstituting an additional warranty. HPEshall not be liablefor technical or editorial errors or

omissionscontained herein.

Microsoft, WindowsServer,and Windowsareeither registered trademarks or trademarks of Microsoft Corporation in theUnitedStates

and/or other countries. Intel and Xeon are trademarks of Intel Corporation in theU.S.and other countries.

4AA6-3740ENW,January 2016

Resourcesandadditional links

Toreadmoreabout HPEsolutions for Exchangepleaserefer to http://h17007.www1.hpe.com/us/en/enterprise/converged-

infrastructure/info-library/index.aspx?app=microsoft_exchange

HPESizer for Microsoft ExchangeServer 2013

hpe.com/solutions/microsoft/exchange2013/sizer

HPEBladeSystem

hpe.com/info/bladesystem

HPEServers

hpe.com/servers

HPEStorage

hpe.com/storage

HPENetworking

hpe.com/us/en/networking.html

HPETechnologyConsulting Services

hpe.com/us/en/services/consulting.html

HPEConvergedInfrastructureLibrary

hpe.com/info/convergedinfrastructure

Contact Hewlett PackardEnterprise

http://www8.hp.com/us/en/hpe/contact/contact.html

Tohelp usimproveour documents,pleaseprovide feedbackat hpe.com/contact/feedback.