08Q4 VM Exchange Server 2007 VI3 WP

8/4/2019 08Q4 VM Exchange Server 2007 VI3 WP

1/16

Deploying Exchange Server 2007 on VMware

Infrastructure: A VMware Internal Case Study

W H I T E P A P E R


2/16

2

VMware white paper

Tbl of Contnts

excutiv Summy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3

exchng Sv 2003: Th Lgcy envionmnt nd Its Limittions . . . . . . . . . . . . . . .3

Migtion Divs fo Vitulizd exchng . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Plnning nd rquimnts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Dsign & Implmnttion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Conclusions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15


3/16

VMware white paper

excutiv Summy

VMware is an industry leading technology company. Email and

instant messaging are mission-critical services that providecommunications and messaging-based notifications for

thousands of VMware employees. The Microsoft Exchange-

based messaging systems at VMware support basic email and

integrate with critical workflow tools, such as engineering

notifications (build, release, bug tracking), shared document

repositories (public folders) for internal user communities,

and company-wide help desk functions. Email is a mission-

critical medium for timely communication with key external

communities including customers, partners, analysts and media

contacts.

Before this project, the Exchange environment at VMware was

based on a traditional, non-virtualized architecture, making it

costly and difficult to maintain performance levels, enable high

availability and resiliency against various failure modes, and

manage downtime. Scaling the system and making optimal

use of existing and new hardware technologies also proved

challenging.

By virtualizing their Exchange Server 2007 deployment, VMware

has gained the ability to:

1) Effectively eliminate planned downtime and maintenance/

upgrade windows on every Exchange 2007 Server using the

VMware HA, VMware DRS and VMware vMotion capabilities

of VMware Infrastructure.

2) Execute immediate recovery from host, application andsite failures through a combination of VMware HA and

new clustering technologies (such as clustered continuous

replication), eliminating single points of failure across various

resource/host/application/site components.

3) Make the most efficient possible use of their vir tual

infrastructure by deploying virtual machines at greater

densities with equal or better performance through optimal

utilization of server, storage and network infrastructure.

4) Achieve exceptional scalability in a building block design,

streamlining tasks such as adding user accounts, mailbox

servers and other role servers while using robust virtual

machine capabilities (design, placement, load balancing) toenable highly flexible Exchange services.

5) Accelerate Exchange life cycle and management functions

with virtual machines and templates for rapid testing,

provisioning, along with virtual snapshots and clones to help

with rapid troubleshooting and problem reproduction.

The benefits realized from virtualized deployment of Exchange

Server 2007 are a direct reflection of the VMware IT departments

Virtualize First policy that has resulted in more than 85 percent

of VMware business applications running in virtual machines

today. The companys ultimate goal is to virtualize 100 percent

of its business applications on VMware Infrastructure to realize

similar benefits.

The virtualized Exchange Server 2007 deployment at VMware

supports more than 4 million messages per day, including 1.5

million external messages (spam included) and supports over

7,500 user mailboxes with a Heavy Average User Load Profile.

The deployment allows good headroom for growth in a highly

available and disaster resilient architecture.

The Exchange environment at VMware is deployed on

10 physical servers for all Exchange Server 2007 roles

(approximately 40 virtual machines) in a multisite configuration.

In comparison, the traditional non-virtualized deployment

of Exchange Server 2003 required 14 physical servers at a

single site for just the mailbox server roles. Enhanced end

user experience, server-side throughput and availability of the

Exchange environment are critical requirements that have been

met and exceeded by the deployment.

This project has been an unqualified success for VMware, and

reflects the experience of many VMware customers worldwide

who are also running Exchange on VMware Infrastructure.

This white paper defines the high-level business requirements,

design considerations and planning approach that were

taken by the VMware IT staff, detailing the technical design

and implementation process, and highlighting important

operational policies for backups and availability.

exchng Sv 2003: Th Lgcy

envionmnt nd Its Limittions

The legacy messaging environment at VMware consisted of

Exchange Server 2003 Enterprise Edition in a traditional non-

virtualized deployment. Two highly available failover clusters

using the Microsoft Cluster Service (MSCS) were used to run all

mailbox and public folder servers.

All Exchange Servers and Active Directory infrastructure

servers were deployed in a single-site model. A total of 14

physical servers were required for mailbox roles supporting

approximately 7,500 total mailboxes deployed in a single forest/

domain/site AD topology. All other roles were deployed onvirtual machines.

Details of the Exchange Server 2003 environment are as follows:

Network Infrastructure

1Gigabitpersecond(1Gbps)switchednetworkbetween

all clustered nodes, internal mail clients, and backup

infrastructure

10Megabitpersecond(10Mbps)switchedconnectivityfor

cluster heartbeat communication via unmanaged network

switch


4/16

4

VMware white paper

Active Directory Infrastructure

TheExchange2003organizationwasdeployedinasingle

forest, domain and site configuration (all Active Directoryservers were virtualized)

Exchange Server 2003 Topology

Front-EndServers

Two(2)virtualizedExchange2003serversformultiple

protocols (IMAP, POP3 and HTTP)

Hardwareloadbalancerprovidedevendistribution

between clients and the front-end servers

Back-EndServers

Eightnode(6active/2passive)MicrosoftCluster

o Five (5) mailbox servers for all of North and South

America users

o One (1) mailbox server serves all Asia Pacific/EMEA

users

Six-node(4active,2passive)MicrosoftClusterwithfour

(4) mailbox servers for new hire employees of VMware

Totalof7,500mailboxesonexistinginfrastructure

Client Systems and Protocols in use

MAPI

Outlook2000/2003/2007clients

WebDAV/HTTPprotocol

OutlookWebAccess(OWA)

ActiveSync

Entourage2004/2008(Microsofte-mailclientforMacOS)

Evolution(E-MailclientforLinux)

MiscellaneousIMAPclients

Thunderbird(IMAPemailclientbyMozilla)

Mail.app(emailclientforMacOSXandiPhone)

Evolution(configuredasIMAPclient)

The legacy Exchange Server 2003 native-OS deployment posed

significant limitations:

1. Scalability of the environment was a serious issue. The user

population and email requirements at VMware had grown

rapidly during the life of the system. The original Exchange

Server 2003 environment consisted of 14 physical servers

which could not adequately support 7,500 heavy user

mailboxes.

HeavyIMAPusage,sincecontent-conversiontookplace

on the mailbox servers.

HeavyI/Oandlargemailboxeswereextremelycommon.

Adding more hardware was not cost-effective, due to

expense for acquisition, management and operations, and a

desire to reduce datacenter footprint.

Fgu 1. Lgcy excng Sv 2003 envonmn


5/16

VMware white paper

2. Traditional non-virtualized deployments precluded

consolidation of the mailbox server instances to leverage

larger and more powerful servers. Exchange Server 2003 and

its32-bitarchitectureislimitedto4GBeffectivememory;

it is unable to use modern large memory and cache

effectively. This leads to stability issues that included memory

fragmentation, small and inefficient use of allocated database

cache and kernel exhaustion. In this situation, the only

remedy is a reboot of the Exchange server and downtime for

its users.

3. A single point of failure exists in Single Copy Clusters (SCC)

deployed with SAN storage. Corruption of the information

stores or SAN infrastructure inconsistencies such as improper

zoning procedures, lose of connectivity etc are a significant

vulnerability.

4. Physical server configurations are more difficult to provision

and maintain. Provisioning new hardware to replace older

Exchange Servers is extremely diff icult. Reboots and system

outages were required to complete simple server hardware

upgrades.

5. Production backup schedules were difficult to meet. Backups

took longer as users were added, and had a greater negative

impact on the production systems during normal operation.

6.Storage,clusterorsitefailurewouldrequirerecoveryof

the data from tape, and an extended outage. Tape-based

disaster recovery was adequate as a last resort, but imposed

potentially significant down times that were not acceptable

as an only option.

7. Early in the deployment of Exchange, mailbox quotas were

not enforced. As the companys growth accelerated, lack of

mailbox quotas in the Exchange Server 2003 environment

led to unpredictable server / storage utilization and mailbox

growth.

8. Uneven distribution of mailbox loads required frequent

mailbox movement between the mailbox servers to ensure

proper load balancing and consistent end-user experience.

9. Availability levels were reduced overall because of the

planned application and hardware upgrades, load balancing/

scalability and stability issues.

Migtion Divs fo Vitulizd exchng

The limitations inherent in the legacy Exchange 2003

deployment were significant reasons to take the opportunity to

redesign and redeploy the messaging infrastructure in virtual

machines, in addition to evaluating a transition to Exchange

Server 2007. As it became obvious that Exchange Server 2007

in and of itself could alleviate some scalability issues, VMware

took this opportunity to ensure that running Exchange 2007 on

VMware Infrastructure would offer very significant additional

improvements in performance, scalability, availability, resiliency,

life cycle management while reducing overall costs.

The main drivers toward the virtualized Exchange Server 2007deployment at VMware are listed below:

VMware Virtualize First Policy

VMware follows a Virtualize First policy when evaluating

application deployments and upgrades so as to realize

important benefits enabled by VMware Infrastructure. These

benefits include:

1) Dealing much more effectively with planned downtimes

and maintenance/upgrades by using technologies such as

VMware vMotion and VMware DRS.

2) Immediate recovery from host, application and site failures

combination of using VMware HA capabilities and traditionalclustering technologies (MSCS) to eliminate single points of

failure across various resource/host/application/site levels.

3) Better use of hardware resources VMware Infrastructure

enables greater density of vir tual machines that can optimally

utilize various server/storage/network resource.

4) Highly scalable infrastructure Adding users and mailbox

servers/roles based on VMware Infrastructure is reduced to

a simple building blocks approach based on robust virtual

machine design/placement/load balancing/scaling models.

5) Expedite Exchange Management tasks and accelerate life

cycle virtual machines and templates provide rapid testing,

provisioning;snapshotsandclonescanhelpwithveryrapidtroubleshooting and problem reproduction.

Whileevaluatingtheperformanceand64-bitarchitectural

enhancements in Exchange Server 2007 from an upgrade

perspective, VMware concluded that combining the benefits of

Exchange Server 2007 benefits with the benefits of virtualization

benefits created a winning combination. This design improved

agility for IT Operations by leveraging VMware Infrastructure

capabilities such as vMotion and VMware HA.

Easier Testing and Prototyping

It was very important to accelerate the life cycle of design/test/

stage/production and not be curtailed by the limitations of

traditional non-virtualized deployments. As the initial design

process was completed, testing the design under simulated load

was essential to ensuring that Exchange 2007 would operate as

expected. Testing in a virtual environment enabled very rapid

changes, and effective testing on less hardware.


6/16

6

VMware white paper

Scalability and Server Containment

The demands of the messaging infrastructure at VMware

and heavy mail usage user profile required more computeresources.WhileExchange2007isabletoexploitmodern

large memory architectures more effectively, it still cannot

be designed in accordance with Microsoft best practices in a

non-virtualized deployment that utilizes the entire capability

of a modern server. Using more, smaller servers would avoid

wasting compute resource, but would increase expense for

hardware, administration, datacenter space, power and cooling.

Virtualization creates an opportunity to reduce server footprint

and preserve the option to use more powerful servers later.

Application Stability

The inherent limitations of running Exchange Server 2003 in

a traditional IT environment, along with the VMware HeavyAverage User Profile, resulted in an unstable environment

because of application downtime caused by memory

fragmentation, the needs for planned hardware upgrades and

related performance issues. The business requirement for a

much more stable and inherently scalable environment was a

major factor behind virtualized deployment.

Leveraging64-bithardwarecapabilitiesfullythroughExchange

Server 2007, and enabling better scale up on new and

muchmorepowerful64-bithardwarewerealsoimportant

considerations behind the virtualized deployment.

Enhanced Availability

Maintenance tasks are a fact of life for production servers.

Unfortunately, they also require server downtime. As a

worldwide company, there are no off hours at VMware.

Someone will always notice a mail outage.

WithallExchangeserversinvirtualmachines,everyserver

can be moved while running to allow hardware maintenance,

upgrades or replacements without interruption to the service.

VMware vMotion makes this a simple exercise. Additionally,

VMware DRS will automatically move a virtual machine to an

appropriate destination when a server becomes heavily loaded.

These capabilities require virtualization, and can significantly

reduce outages resulting from single-server issues.

Site Resiliency

VMware required a rapid response to any failure ranging from a

single resource to loss of an entire site. The legacy environment

relied on a single-site design with a tape-based recovery

process. The combination of VMware HA and the application

failover capabilities built into Exchange Server 2007 CCR can

effectively eliminate single points of failure inherent in a non-

virtualized implementation, and assure a rapid return of services

in scenarios less dire than a true regional catastrophe.

Plnning nd rquimnts

Key Planning Considerations and Steps

VMware IT Operations teams began a proof of concept of

Exchange 2007 on a fully-virtualized platform in October 2007.

Detailed planning started in January 2008, migrations began

March 2008 and the entire environment was fully virtualized and

complete by June 30, 2008.

VMware IT staff enlisted assistance from various groups within

VMware, including Performance Monitoring, Quality Assurance,

Product Engineering and the VMware Business Solutions unit.

Withthecooperationofthevariousteams,VMwareensuredthat

the planning phase resulted in a solid design framework that

would meet or exceed clearly defined business requirements.

The approach VMware took in the planning phase was as

follows:

1. Establish a baseline from the existing Exchange 2003

deployment to measure its capabilities and limitations across

dimensions such as availability, per formance, scalability,

disaster recovery, and backups/restore.

2. Clearly define the overall workload for the Exchange 2007

deployment, and factor in key assumptions for scaling,

growth and headroom.

3. Collect and rationalize commercial and operational

requirements from across the company.

4. Translate and map the business requirements for scaling,

availability, recoverability, backups, and other parameters intoa set of architectural and design plans.

5. Size the infrastructure components, including servers, storage

and networking.

6.Developatestenvironmentcomparabletotheproposed

architecture,includingVMwareESXhosts,virtualmachines,

storage sizing and network topology.

7. Validate the test environment using Exchange Server 2007

testingutilities(LoadGen,JetStress,ExBPA).Analyzeand

remediate any observed anomalies, adjust the design as

necessary and iterate through the tests.

8. Create a migration plan based on Business and Technical

requirements using a pilot-to-beta-to-production approach.

9. Build the production environment based on the final

validated test environment.

10. Transition existing Exchange 2003 mailboxes to virtualized

Exchange Server 2007 production environment.

11. Monitor new production environment for anomalies, testing

and correcting as necessary.


7/16

VMware white paper

Business Requirements

VMware places very high demands on its messaging systems

andrequiresthattheybeavailable24x7x365.Asaresult,VMware had the following enhanced business requirements as

it planned the migration over to the new Exchange Server 2007

messaging environment:

1. Multi-Site Resiliency Provide a robust, reliable, resilient and

mobile virtualized messaging infrastructure that is capable

of withstanding site failure. Ensure that the infrastructure

enables fast recovery from outages due to server or

information store outages or corruption. Ensure the ability to

maintain two (2) synchronized copies of the Exchange Server

2007 data.

2. High Availability Elimination of any single point of failure

in the architecture. Ensure that the system is capable ofimmediate failover of the messaging systems with minimal

to zero IT interaction and without significant impact to

connected clients.

3. Improved Performance Reduce latency for connected

clients. Ensure that significantly improved I/O performance

in Exchange Server 2007 translates to a better end-user

experience with faster Outlook response times and reduced

RPC latency.

4. Scalability Use VMware Infrastructure to increase hardware

utilization efficiency, and to overcome the Exchange 2007

eight-core,32GBmaximumrecommendedconfiguration.

5. Zero-downtime upgrades Use VMware vMotion alongwith Microsoft Clustering services to enable patching and

upgrading with minimal impact to end users.

6.QuotaImplementationControlmailboxsizeandgrowthto

allow predictable utilization of storage resources and reduce

administrative overhead associated with mailbox movement.

7. Flexible backup options - Provide flexible options around

backups (with VMware Consolidated Backup, guest backups,

array-based snapshots and passive node backups)

8. Support for multiple protocols A significant percentage of

the VMware mail users connect to the Exchange environment

via IMAP, POP, and HTTP clients.

9. Reduce overall costs associated with operating and growing

our user base/messaging needs, have a much more

inherently scalable operational foundation.

10. Continue to support a very Heavy Average Mail User profile.

Design Considerations

Architectural Enhancements in Exchange Server 2007

ExchangeServer2007isdesignedtoexploit64-bitarchitectures.A brief list of advantages (more exhaustive information can be

found at http://www.microsoft.com/exchange) include

Abilitytouse64-bitmemoryaddressesscalesbetter

with more memory - mailbox sizes and the number of user

accounts per server increase.

Reducedinput/output(I/O)requirementsenabledbythe

largermemorycachesavailableon64-bitsystemsalsoleads

to better CPU scaling.

TypicallyresultsinmuchbetterI/Operformanceand

reduction in I/O requirements, which translates to more

optimal use of storage.

Leveraging VMware Infrastructure Capabilities

The ability to scale Exchange mailbox servers on one physical

server was a major driver for vir tualization of the Exchange

infrastructure.WithVMwareInfrastructure,theVMware

IT organization can utilize larger physical servers running

VMwareESX3.5tohostmultiplemailboxservers,aswellas

peripheral roles (hub transport, client access, etc). Stacking

multiple virtual machines on powerful physical servers enables

better hardware efficiency and utilization while helping to

overcome the Microsoft recommended Exchange maximum

ofeightprocessorcoresand32GBofRAM.Deploymentof

new Exchange 2007 virtual machines from preconfigured

templates significantly reduced the time to provision. VMware

HA and VMware DRS enable the required up time during both

scheduled and unscheduled downtime for host patching and

upgrades.

Very High Resiliency - Application Aware Clustering

The dependency of VMware on messaging as a mission-critical

application, and resulting high availability requirements,

drove the decision to continue use of Microsoft clustering

technologies in conjunction with VMware HA.

VMware HA minimizes the duration of any outage by

automatically restarting all virtual machines on another VMware

ESXserverintheeventofserverhardwareoutage.Whileclustering can improve availability for mailbox servers, VMware

HA protects all Exchange roles, improving overall availability

when compared to a physical deployment. VMware HA

minimizes any disruption to the messaging environment, and

can be easily tested to ensure that the environment recovers

as expected. VMware HA is simple to set up and protects every

virtual machine without requiring complex clustering software.

VMware HA was chosen to address the majority of cases

where simple hardware failure in a non-clustered server can

compromise overall operation.


8/16

8

VMware white paper

Exchange 2007 Cluster Continuous Replication (CCR) is a

high availability feature of Microsoft Exchange Server 2007

that combines the asynchronous log shipping and replay

technology built into Exchange 2007 with the failover and

management features provided by the Cluster service

(http://technet.microsoft.com/en-us/library/bb124521.aspx).

Exchange CCR removed the architectural dependency on

shared storage and eliminated exposure to a complete site

outage as a single point of failure (a core requirement of the

design). Exchange 2007 CCR on top of VMware HA allows

VMware to have application-aware high availability, as well as

host-level high availability. VMware HA automatically restarts

virtualmachinesonotherVMwareESXhostsinavirtualcluster

intheeventofanESXhostfailure.

Utilize Multicore Capabilities

Exchange deployments on physical servers are not able to

effectively utilize the compute power now available in more

powerfulquad-core(andmore)systems.VMwareESXhelps

to resolve this issue and enables significantly more scalability

for the virtualized Exchange 2007 environment. VMware

has documented (http://www.vmware.com/solutions/

business-critical-apps/exchange/resources.html ) optimized

deployment models and best practices based on a building

block approach.

The building block VMware IT utilizes is based on four vir tual

CPUs. This is based on the 500 heavy-user-per-core benchmark

which was achieved both in-house and externally. VMware

decided on the use of blade technology to reduce the physical

space and cooling requirements that were placed on the

datacenter. The choice of more powerful blades allows the

infrastructuretoscaleandreducesoverallTCO.Having16

cores available in a single blade server enables deploying up

to four active mailbox servers per blade and still leaves plenty

of resources/headroom available to support passive mailbox

servers and peripheral roles (hub transport, client access

servers, etc.).

64GB of Memory per Blade Server

Using64GBofmemoryperbladeserverenablesmaintaining

lowercostbyutilizingsmaller4GBmemorymodules.The

VMwareITmailboxserverbuildingblockutilizes12GBof

memory(2,000usersat5MBmailboxcacheand2GBforthe

operatingsystem);allotherperipheralrolesareallocated

4GBofmemoryeach.Thesebuildingblockseasilysatisfythe

requirements of the four active mailbox servers and can still

accommodate the various peripheral roles that may run on

anyonephysicalVMwareESXhostatonetime.

Storage Considerations

Whendesigningthestoragesolutionforthevirtualized

Exchange 2007 environment, several factors were taken intoconsideration:

Virtualdiskvs.rawdiskmappingInternaltesting

confirmed that VMware could meet I/O requirements

regardless of whether virtual disks or raw disk mappings

were used. In some cases, virtual disks were slightly

faster. Raw device mappings in physical compatibility

mode was selected to accommodate array based

clones or snapshots. This approach could also be

facilitate swinging LUNS back to physical servers for

troubleshooting. Note that with the release of VMware

Consolidated Backup 1.5, which supports VSS, virtual disks

can be used to deliver consistent and complete backups

of virtual machines via snapshots.

PerformanceDisk/spindlecountisasignificantissue

that relates directly to the balance between designing for

capacity versus throughput. Disk input/output rate (I/O

per second, or IOPS) are a key design consideration for

Exchange. IOPS demands may effectively prevent the use

of fewer high-capacity drives, or require that those large

drives be only partially utilized.

Forexample,300GBdrivesprovidehighcapacity,but

reducetheavailableIOPSperGBofdata.Agroupof

4x300GB15krpmdrivescandeliverapproximately

500600IOPSfor1,200GB.Choosing146GBdrives

insteaddeliversthesameIOPSratefor584GB.IftheI/O

requirementperGBislow,thelowercostperGBoption

might be appropriate. On the other hand, if the I/O

requirementperGBishigh,systemperformancewillbe

seriously degraded unless the smaller drives are selected

or the larger drives are only partially used, wasting the

extra capacity. This type of trade-off highlights the

importance of working closely with the storage vendor

of choice to determine optimal disk and I/O capabilities

based on performance and space requirements.

Disklayout&quotasThestorageconfigurationconforms

to EMC and Microsoft best practices. The design utilizes

RAID10 for all database and log LUNs, and ensures physicalseparation between the two to ensure recoverability

and optimum performance. Each site has a single EMC

CX3-80SANStoragearrayforstorageofthevirtual

machines and all local Exchange data. In the legacy

VMware messaging environment, user mailbox quotas

were not enforced. Several mailboxes had grown past

10GBinsize.Thisrequiredmoreadministrativeeffortto


9/16

VMware white paper

maintain manageable Exchange Server information stores

by constantly moving mailboxes. The implementation of

user mailbox quotas mitigates this additional administrative

effort.Standardquotasof2GBand5GBareenforcedacross

all mailboxes. The majority of the user population maintains

amailboxbelow2GB.Userswithspecialcircumstancesor

business justifications for an increased mailbox size can obtain

5GBofmailboxstoragewiththeappropriateapprovals.No

mailboxesabove5GBwillbesupported.

Dsign & Implmnttion

WiththetransitiontoExchangeServer2007,VMwarehas

achieved a consolidation ratio of about four to one (4:1) with

approximately 40 virtual machines running on 10 physical

servers.Toplacethisintheproperperspective;10physical

servers run the entire virtualized Exchange Server 2007

organization including the domain controllers, anti-spam and

perimeter e-mail routing servers. The legacy Exchange Server

2003 environment had 14 mailbox servers alone.

The Exchange 2007 environment is 100 percent virtualized on

VMwareESXv3.5.ThevirtualmachinesrunWindows2003x64

R2 SP2, Enterprise Edition.

Exchange Server High Availability

To meet the business requirements of no single point of failure,

automated failover and multiple synchronized copies of the

mailbox data and site resiliency, VMware combines VMware

HA with a new Microsoft Exchange Server 2007 application-level high availability replication technology known as Cluster

Continuous Replication (CCR). VMware HA will restart any virtual

machine from a failed physical node on a running node in the

HA/DRS cluster. CCR affords the ability to replicate the Exchange

server data within the datacenter known as in datacenter

service or to a remote site datacenter known as site resiliency.

One HA/DRS cluster is deployed in both of the VMware

datacenters. The CCR design used at VMware populates each

HA/DRS clusters with eleven CCR cluster-nodes. In the event

ofanunscheduledESXhostoutage,theCCRclusterwill

automatically move the clustered mailbox role to the passive

node in the second datacenter and continue to provide email

services. Due to the VMware HA cluster, the failed active cluster-

nodewillberestartedonanyremainingavailableESXhost,and

CCR will initiate reverse replication to ensure that the cluster

is in sync. The same methodology would be used in the case

of a total site failure. To achieve complete autonomy between

datacenters, a third site was chosen to house the file-share

witnesses used by the CCR clusters to maintain quorum.

Utilizing the VMware Infrastructure Feature Set Optimally

To further improve upon the new design, the following

capabilities within the VMware Infrastructure feature set wereutilized:

VMware High Availability (HA) Allows virtual machines to

berestartedonothernodesofaVMwareESXServerclusterin

the event of hardware failure. VMware HA is utilized to ensure

the virtual machines are powered on in the case of a server

hardware failure.

Scenario:

Hostserverfails;thiscausestheclustertofailovertothe

remote site.

VMwareHArestartsallExchangevirtualmachinesonanother

host in the local cluster.

Replicationcannowreversedirectiontoprimarysiteupdate

to synchronize the Exchange. database.

FailbackcanoccurmanuallywhentheExchange

Administrator is available.

VMware Distributed Resource Scheduling (DRS) VMware

DRS continuously monitors utilization across resource pools

and will intelligently allocate resources to virtual machines that

require additional resources based on pre-defined policies.

VMware DRS is used with ful l automation level for all Exchange

server roles including clustered mailbox servers to allow for

auto-placement based on resource requirements.

Scenario:

ExchangeMailbox,CASorHUBserverorserversrequire

additional compute resources

VMwareDRSscanstheclustertodeterminethehostthatcan

best satisfy the request for additional resources.

Virtualmachineswillbemigratedtothehostinthecluster

that will satisfy the request using vMotion.

ThisresultsinahighlybalancedandoptimizedExchange

configuration.

VMware vMotion Allows virtual machines to move from one

physical server to another physical server with no impact on

operation or end user services. VMware vMotion is used to movevirtual machines between physical servers within the VMware

HA cluster for physical server maintenance and patching. All

of the virtual machines in the environment (including active

mailbox servers) can be migrated under load with vMotion.


10/16

10

VMware white paper

Scenario:

AVMwareESXhostneedstobeupdatedviapatchesand

additional memory must be added to the host. AffectedESXhostcanbeplacedinMaintenancemode.

VirtualmachinesmigratetootherESXhostsinthecluster

freeing up the affected host.

ESXhostispatchedandadditionalmemoryisadded.

ESXhostisrestartedandvirtualmachinesaremigratedback

to upgraded host.

Mail Routing and Client Access

VMware uses a centralized infrastructure and administration

model. In this model, the majority of infrastructure services

are served from two datacenters near the companys main

campus in Palo Alto, California. The two-datacenter designof the Exchange 2007 infrastructure and the nature of mail

routing in Exchange 2007 led to an interesting dilemma - one

or two Active Directory sites?

One Active Directory site would allow for simplicity, fewer

domain controllers, fewer overall Exchange servers and

fewer customizations. However, it was possible that mail

between two mail users whose mailbox resided on the

same mailbox database could potentially bounce between

the Palo Alto and Santa Clara datacenters. Two Active

Directory sites would allow for all mail local to Palo Alto

or Santa Clara to stay within their respective datacenters

usinglocalhubtransportservers.Wechosetouseatwo

Active Directory site configuration.

The two Active Directory design also meant changes to

how client access worked. In this design, all connections

toSantaClaramailboxesviaOWA,ActiveSync,andOutlook

Anywhere are proxied to Santa Clara through Palo Alto.

IMAP and POP connections are also proxied through Palo

Alto, which required customization of the default IMAP andPOP Exchange Server 2007 Environment

Fgu 2. Cln ccss nd ml oung VM


11/16

1

VMware white paper

Exchange Server 2007 Environment

The new Exchange Ser ver 2007 environment runs entirely

in virtual machines on VMware Infrastructure 3. The virtualmachines are deployed as 22-mailbox cluster nodes that make

up 11 Cluster Continuous Replication (CCR) clusters. This houses

11 highly available Exchange Server 2007 clustered mailbox

servers (CMS). The cluster nodes are geographically dispersed

between the Palo Alto and Santa Clara datacenters. This design

allows for site resiliency and eliminates single point of failure.

Each site has dedicated hub transport, client access and domain

controller/global catalog servers. To efficiently route traffic

between the various transport servers, a second Active Directory

site was created for Santa Clara. The addition of this site helps

avoid mail from unnecessarily being sent across the metro link.

Storage Subsystem Architecture/Design

The storage configuration leverages our storage vendors best

practices. The design utilizes RAID10 for all database and log

LUNs, and ensures physical separation between the two to

ensure recoverability and optimum performance. Each site has

asingleEMCCX3-80SANStoragearrayforstorageofthevirtual

machines and all local Exchange data.

DatabaseLUNsareconfiguredusingfour(4)300GB/15kFC

spindles. Each database LUN houses two (2) mailbox databases

for a maximum of 350 users. The standard mailbox server

building block uses seven (7) database LUNs configured as

physical mode RDMs. The four spindle LUN allows us to satisfy

our 1.15 per mailbox I/O requirement.TransactionlogLUNSareconfiguredusingten(10)300GB/15k

FC spindles. The single log LUN is configured as a physical mode

RDM and is shared between the 14 storage groups that reside

on the standard mailbox server building block.

System/boot partitions use virtual disks on VMware VMFS

volumes configured for RAID5 at the array level.

Fgu 3 Vul Mcn Dsk Lyou


12/16

12

VMware white paper

Performing Backups in the Exchange Server 2007

Environment

Daily backups are performed using traditional Exchangestreaming backups to virtual tape libraries (VTL). The use of VTLs

allows VMware IT to keep 30 days worth of backup data online

and quickly accessible. Due to the tape-free design, there is no

risk of having tapes off-site when needed or having to locate all

of the different tapes used for a particular save-set. This allows

for the implementation of a less stringent backup schedule

such as weekly full and daily incremental (or differential).

Furthermore, recovery from alternate backup media would only

be required in the event of multiple failures, one at the primary

(active) data source and one at the alternate (passive) data

source.

Fgu 4. Bcku envonmn

Fgu 5. Bcku Scdul

Sun Mon Tues Wed Thurs Fri Sat

SG1,8 Full inc inc inc inc inc inc

SG2,9 inc Full inc inc inc inc incSG3,10 inc inc Full inc inc inc inc

SG4,11 inc inc inc Full inc inc inc

SG5, 12 inc inc inc inc Full inc inc

SG6, 13 inc inc inc inc inc Full inc

SG7, 14 inc inc inc inc inc inc Full


13/16

1

VMware white paper

Hardware Configuration

Fgu 6. hp c7000 Bld Csss

Storage Area Network Configuration

Fgu 7. SaN Confguon


14/16

14

VMware white paper

Fgu 8. VM eSX Svs/excng insucu Vul Mcn

Fgu 9. excng Sv 2007 Clus nd S Confguon


15/16

1

VMware white paper

Conclusions

VMware has successfully deployed a fully virtualized Exchange

Server 2007 environment on VMware Infrastructure. This

deployment has met and exceeded the business and technical

requirements that were defined and which have been outlined

in this paper. The benefits of virtualization combined with the

benefits of Exchange 2007 offer capabilities and cost savings

that are not otherwise available.

By virtualizing its Exchange infrastructure, VMware has gained

the ability to:

1) Easily scale up and continuing to grow the deployment as the

companys base of users grows.

2) Deal eff iciently with discontinuous points of growth, such

as acquisitions and rapid user group provisioning/data

migrations.3) Continually monitor and optimize the physical and virtual

infrastructure to continue to drive down costs associated with

space, power/cooling and enabling the transition to green IT.

4) Evaluate and leverage advancements in hardware

technologies with minimal/zero disruption on the production

Exchange environment.

This project has been a complete success, and reflects the

consistently positive experience that VMware customers report

after deploying Exchange 2007 on VMware Infrastructure.

Fgu 10. excng Sv 2007 toology


16/16

VMware, Inc. 3401 Hillview Ave Palo Alto CA 94304 USA Tel 877-486-927 3 Fax 650-427-50 01 www.vmware.com

Copyright 2008 VMware, Inc. All rights reserved. Protected by one or more U.S. Patents Nos. 6,075,938, 6,397,242, 6,496,847, 6,704,925,

6,711,672, 6,725,289, 6,735,601, 6,785,886, 6,789,156, 6,795,966, 6,880,022, 6,944,699, 6,961,806, 6,961,941, 7,069,413, 7,082,598, 7,089,377,

7,111,086, 7,111,145, 7,117,481, 7,149,843, 7,155,558, 7,222,221, 7,260,815, 7,260,820, 7,269,683, 7,275,136, 7,277,998, 7,277,999, 7,278,030,

7,281,102, 7,290,253, 7,356,679, 7,409,487, 7,412,492, 7,412,702, 7,424,710, 7,428,636, 7,433,951, 7,434,002 and patents pending. VMware is

a registered trademark or trademark of VMware, Inc. i n the United States and/or other jurisdictions. All other marks and names mentioned

herein may be trademarks of their respective companies.

08Q4 VM Exchange Server 2007 VI3 WP

Documents

Transcript of 08Q4 VM Exchange Server 2007 VI3 WP