© 2010 VMware Inc. All rights reserved

Enterprise Java Applications on VMware

Technical Workshop

Emad Benjamin – VMware Staff Solutions Architect on Java and vFabric

2

About the speaker

I have been at VMware for the last 7 years, working on Java and

vSphere

20 years experience as a Software Engineer/Architect, with last 15

years focused on Java development

Open source contributions

Prior work with Cisco, Oracle, and Banking/Trading Systems.

Authored the Enterprise Java Applications Architecture on VMware

BCA1950

Production Proven Methods

of Running Enterprise Java

on vSphere.

Emad Benjamin, Staff Solutions Architect on Java and

vFabric, VMware, Inc.

4

Disclaimer

This session may contain product features that are

currently under development.

This session/overview of the new technology represents

no commitment from VMware to deliver these features in

any generally available product.

Features are subject to change, and must not be included in

contracts, purchase orders, or sales agreements of any kind.

Technical feasibility and market demand will affect final delivery.

Pricing and packaging for any new technologies or features

discussed or presented have not been determined.

5

Agenda

Conventional Enterprise Java Platforms

Enterprise Java Platform Architecture on VMware vSphere™

Design and Sizing

Best Practices

High Availability

Migration

Performance

Troubleshooting

Questions

6

Conventional Enterprise Java Platforms

7

Conventional Enterprise Java Platform

Enterprise Java applications are multitier

Multitier complexity leads to cascade impact when changes are

introduced

Multi-organizational challenges lead to increased cost

DB Servers Java Applications

Load Balancer Tier

Load Balancers Web Servers

IT Operations

Network Team

IT Operations

Server Team

IT Apps – Java

Dev Team

IT Ops & Apps

Dev Team

Organizational Key Stakeholder Departments

Web Server Tier Java App Tier DB Server Tier

8

Enterprise Java Platform Architecture

on VMware vSphere

9

Enterprise Java Applications Platform Architecture on vSphere

SHARED, ALWAYS-ONINFRASTRUCTURE

SHARED INFRASTRUCTURE SERVICES

Capacity On Demand High AvailabilityDynamic

APPLICATION SERVICES

DB ServersJava ApplicationsLoad balancers Web Servers

VMware vSphere

High Uptime, Scalable, and Dynamic Enterprise Java ApplicationsLoad Balancers as VMs

Web Servers

Java Application Servers

10

VMware vFabric Cloud Application Platform

Dynamic Load

Balancer

Performance

Management

Policy-driven

Automation

Elastic

App Server Cloud

Messaging

Global Data

Management

Application Infrastructure for the Cloud Generation

Virtual Datacenter Cloud Infrastructure and Management

tc Server ERS (Apache) Hyperic RabbitMQ GemFire

Frameworks & Tools

vFabric Platform Services

Rich Web Integration Batch Data Access Social /Cloud Tool Suite

11

Choice of frameworks

.js

12

Data

Services

Other

Services

Msg

Services

Choice of application services

.js

13

Private Clouds

Public Clouds

Micro Clouds

Choice of clouds

.js

Data

Services

Other

Services

Msg

Services

.COM

…

14

Cloud Foundry – key use cases

New applications

Dev-test-trial SaaS extensibility

App modernization

15

Cloud Foundry – key audiences

Grassroots Developers “No one between you and your application”

Friction-free way to develop, test and deploy

Open ecosystem of frameworks and services

Use it your way – open source, micro clouds, choice of public clouds

Self-service access to cloud-scale application resources

SaaS ISVs

“Cloudify your apps”

Elastic programmability and extensibility to your SaaS service

High productivity frameworks and application services

16

Cloud Foundry – under the covers

Router

Infrastructure

Health Manager

DEA: execution environment Services: core, ecosystem,

etc.

user apps Cloud Controller user apps

vmc client STS plugin browser (user app

access)

17

Cloud Foundry – key audiences

Create app, update app, control app

vmc push [appname] [--path] [--url] [--instances N] [--mem] [--no-start]

vmc update <appname> [--path PATH]

vmc stop <appname>

vmc start <appname>

vmc target [url]

Update app settings, get app information

vmc mem <appname> [memsize]

vmc map <appname> <url>

vmc instances <appname> <num | delta>

vmc {crashes, crashlogs, logs} <appname>

vmc files <appname> [path]

18

Cloud Foundry – key audiences

Deal with services, users, and information

vmc create-service <service> [--name servicename] [--bind appname]

vmc bind-service <servicename> <appname>

vmc unbind-service <servicename> <appname>

vmc delete-service <servicename>

vmc user, vmc passwd, vmc login, vmc logout, vmc add-user

vmc services, vmc apps, vmc info

19

Design and Sizing

20

Design and Sizing of Enterprise Java Applications on vSphere

Step 1 –

Establish Load profile

From production logs/monitoring reports measure:

Concurrent UsersRequests Per SecondPeak Response TimeAverage Response Time

Establish your response time SLA

Step 2

Establish Benchmark

Iterate through Benchmark test until you are satisfied with the Load profile metrics and your intended SLAafter each benchmark iteration you may have to adjust the Application Configuration Adjust the vSphereenvironment to scale out/up in order to achieve your desired number of VMs, number of vCPU and RAM configurations

Step 3 –

Size Production Env.

The size of the production environment would have been established in Step2, hence either you roll out the environment from Step-2 or build a new one based on the numbers established

Step 2 – Establish Benchmark

DETERMINE HOW MANY VMs Establish Horizontal Scalability

Scale Out Test How many VMs do you need to meet your Response Time SLAs without reaching 70%-80% saturation of CPU? Establish your Horizontal scalability Factor before bottleneck appear in your application

Scale Out Test

Building Block VM Building Block VM

SLA OK?

Test complete

Investigate bottlnecked layer Network, Storage, Application

Configuration, & vSphere

If scale out bottlenecked

layer is removed, iterate scale out

test

If building block app/VM config problem, adjust

& iterate No

Building Block VM

ESTABLISH BUILDING BLOCK VM Establish Vertical scalability

Scale Up Test Establish how many JVMs on a VM? Establish how large a VM would be in terms of vCPU and memory

Scal

e U

p T

est

Building Block VM

22

Design and Sizing HotSpot JVMs on vSphere

JVM Max Heap -Xmx

JVM Memory

Perm Gen

Initial Heap

Guest OS Memory

VM Memory

-Xms

Java Stack

-Xss per thread

-XX:MaxPermSize

Other mem

Direct

native

memory

Non

Direct

Memory

Virtual

Address

space

23

Design and Sizing of HotSpot and jRockit JVMs on vSphere

Guest OS Memory approx 1G (depends on OS/other processes)

Perm Size is an area additional to the –Xmx (Max Heap) value and

is not GC-ed because it contains class-level information.

“other mem” is additional mem required for NIO buffers, JIT code

cache, classloaders, Socket Buffers (receive/send), JNI, GC internal

info

If you have multiple JVMs (N JVMs) on a VM then:

• VM Memory = Guest OS memory + N * JVM Memory

VM Memory = Guest OS Memory + JVM Memory

JVM Memory = JVM Max Heap (-Xmx value) + JVM Perm Size (-XX:MaxPermSize) +

NumberOfConcurrentThreads * (-Xss) + “other Mem”

24

Design and Sizing of IBM JVM on vSphere

JVM Max Heap -Xmx

JVM Memory

Perm Gen

Initial Heap

Guest OS Memory

VM Memory

-Xms

Java Stack

-Xss per thread

-XX:MaxPermSize

Other mem

Perm Gen

25

Design and Sizing of jRockit JVM on vSphere

JVM Max Heap -Xmx

JVM Memory

Perm Gen

Initial Heap

Guest OS Memory

VM Memory

-Xms

Java Stack

-Xss per thread

Other mem

Metadata Dynamically managed by JVM

26

Sizing Example

JVM Max Heap -Xmx

(4096m)

JVM Memory (4588m) Perm Gen

Initial Heap

Guest OS Memory

VM Memory (5088m)

-Xms (4096m)

Java Stack -Xss per thread (192k*100)

-XX:MaxPermSize (256m)

Other mem (=217m)

500m used by OS

set mem Reservation to 5088m or set to Active

memory used by VM which could be lower.

27

Sizing The Java Heap

JVM Max Heap -Xmx

(4096m)

Eden Space

Survivor Space 2

Old Generation

Survivor Space 1

Slower

Full GC

Quick

Minor GC YoungGen -Xmn

(1350m)

OldGen 27460m

28

But how did we establish –Xmx is 4096MB?

One answer is that we know from testing the application

Alternately: we started the Java application without –Xmx or –Xms

being specified, and then monitor the heap usage during a load

test.

• You are determining the actual active data used in OldGen after a fullGC

• Observe whether you are getting OldGen or PermGen OutOfMemory Errors

Above approach helps you determine live memory “LiveData”

• LiveData is equivalent to OldGen space occupancy after FullGC

• As a general rule size –Xmx (the Heap) => 3 to 4 times the “LiveData”

• PermGen is 1%-6% of –Xmx, or 1.5 times the PermGen after FullGC

• YoungGen (-Xmn) approx 33% of –Xmx, or 1.5 times LiveData after FullGC

29

But how did we establish –Xmx is 4096MB?

2011-10-15T19:59:09.895-0600:

[Full GC

[PSYoungGen: 958,034k-> 0k (1,254,400k)]

[ParOldGen: 1,494,122k-> 1,032,888 k(2,398,095k)]

[ 2,452,156k -> 1,032,888k (3,669,995k)]

[PSPermGen: 113,365k-> 113,365 k(163,840k)],

0.874769secs]

Young Gen

(-Xmn)

Old Gen

(-Xmx “ –” -Xmn)

Perm Gen size

(-XX:MaxPermSize)

Young Gen

before GC

Old Gen

before GC

Perm Gen size

before GC

Full GC

duration

Old Gen

after FullGC

active data “LiveData”

1GB

30

But how did we establish –Xmx is 4096MB?

Note: numbers before “->” are before GC, and numbers after “->”

are after GC numbers.

Since Active Data was approx 1GB, and applying the rule of Heap (-

Xmx) should be 3 to 4 times of Active data size, hence chose Heap

of 4096MB in our sizing example.

Then we applied all of the other relative sizing rules to establish the

other memory segments that we already covered.

Then we used an amount of 5088MB as Memory Reservation for

the VM when all the various memory segments were added up.

31

Sizing Example for a 4096MB Heap and 5GB Reserved VM Mem

JVM Max Heap -Xmx

(4096m)

JVM Memory (4588m) Perm Gen

Initial Heap

Guest OS Memory

VM Memory (5088m)

-Xms (4096m)

Java Stack -Xss per thread (192k*100)

-XX:MaxPermSize (256m)

Other mem (=217m)

500m used by OS

set mem Reservation to 5088m or set to Active

memory used by VM which could be lower.

32

Design and Sizing:

Understanding ESX Memory

Management

33

Why did we use Memory Reservation?

Memory Reservation

• We used an amount of 5088MB as Memory Reservation for the VM when all

the various memory segments were added up.

• This is the physical ESX Host memory guaranteed to be available to the VM

up-on startup such that memory over commitment is avoided

• Hence ESX memory management techniques such as Ballooning and

Swapping avoided in order to preserve performance

34

ESX Memory Management

Guest virtual memory is mapped to Guest Physical is in turn mapped to

ESX Physical Memory

35

ESX Memory Management

The below shows an over committed memory situation, only 4GB

physical is available but 6GB has been allocated to the VMs

ESX uses Transparent Page Sharing (TPS), Ballooning, and host

swapping, in order to support memory reclamation in an overcommitted

situation

36

Understand ESX Memory Management

ESX uses Transparent Page Sharing (TPS) – when multiple VMs are

running it is highly likely that common memory segments exit, for

example Guest OS used is common amongst the VMs, and hence can

be TPS-ed

37

ESX Memory Management

TPS scans both VM and Host memory for common memory segments

to cache

Scan rate controlled by Mem.ShareScanTime, Mem.SharerateMax, and

Mem.ShareScanGHz

38

ESX Memory Management

Ballooning – makes the Guest OS aware that ESX Host memory is short

Due to VMs isolation, the Guest OS is not aware that it is in a VM and is

not aware of the state of other VMs, or the ESX Host memory situation

Balloon driver is loaded into Guest OS, communicates with ESX via

private channel, and ESX will instruct it to inflate (allocate Guest OS

Physical Pages) by an amount. An amount that it needs to reclaim.

39

ESX Memory Management

ESX Host Swapping – this is a last resort, and things are quite bad at

this stage. If TPS and Ballooning didn‟t work, then ESX will swap VM

memory to swap file.

40

Use ESX to Share Memory Between VMs Running Java

Elastic Memory for Java (EM4J)

Description Benefits

• Higher consolidation through memory over-commit for

Java workloads

• Protect Java applications against workload spikes

without wasting memory • Lets you give Java more memory than it needs under

normal load. Memory your application doesn‟t need is

returned to ESX but remains available in case it is

needed to avoid Java out-of-memory errors

• Use virtualization to

overcome the

limitation of Java‟s

static heap

• New „memory balloon‟

runs inside JVM to

capture unused

memory and return it

to ESX

• Memory returned to

JVM when it is under

pressure

41

Design and Sizing:

Undersating GC

42

Which GC?

ESX doesn’t care which GC you select, because of the degree of

independence of Java to OS and OS to Hypervisor

43

Which GC?

Either you tune for Throughput or Latency, one at the cost of the

other

Start with Throughput collector, if latency is not met then move to

Concurrent collector

• Throughput Collector -XX:+UseParallelOldGC enables both multithreaded

young generation and old generation collector

• If -XX:+UseParallelOldGC not available then you can use –XX:UseParallelGC,

but it only allows for multithreaded young generation.

• When using –XX:UseParallelGC the old generation garbage collector used

with this option is the serial collector which is single threaded.

• If you find latency is an issue, then you can move to the CMS collector.

44

When tuning GC use GC logs

To investigate use GCLogging

49.09:

[GC

[PSYoungGen: 958,034k-> 0k (1,254,400k)]

[ 2,452,156k -> 1,032,888k (3,669,995k), 0.108 secs]

[Times: user=1.83 sys=0.01, real=0.11 secs0.874769secs]

-XX:+PrintGCTimeStamps –XX:+PrintGCDetails –Xloggc:<filename>

45

What to measure when tuning GC?

General approach to investigating latency

• Determine Minor GC duration

• Determine Minor GC Frequency

• Determine worst Full GC duration

• Determine worst Full GC frequency

Minor GC measurements drive Young Generation refinements

Full GC measurements drive Old Generation refinements

The decision to switch to –XX:+UseConcMarkSweepGC

• If throughput collector's worst case FullGC duration/frequency compared with

app latency

46

When Tuning Young Generation

Minor GC too

long

Tune MinorGC

(start)

Reduce Young Gen size

Minor GC

freq. high

Increase Young Gen size

yes

No

yes

Measure Minor GC duration

Reduce OldGen size by the same amount as you reduced

YoungGen

Measure Minor GC

Freq.

Adjust Young Gen size to: (TargetFreq/CurrentFreq) *

CurrentYoungGenSize -Also adjust OldGen by same amount

47

Tuning Example Adjust Duration of Minor GC

Java –Xms4512m –Xmx4512m –Xmn1489m –XX:MaxPermSize=96m

Avg GC Minor GC Freq = 2 seconds

Avg Minor GC Duration = 0.05 seconds, ie 50msec

App latency requirements is 40mseconds

• Measured latency 54msec, hence duration too long MUST REDUCE

YoungGen

• Reduce Young Gen by 10% and heap by the same amount

Java –Xms4060m –Xmx4060m –Xmn1340m –XX:MaxPermSize=96m

48

Tuning Example Adjust Frequency of Minor GC

Average Minor GC Freq = 2 seconds

But supposed instead of pause time, the app cares more about

frequency of GC

• Frequency Requirements is 5seconds

• Increase –Xmn (YoungGen) to reduce GC frequency to 5seconds, 5/2=>2.5

• Original –Xmn is 1489, adjusted value is 2.5*1489 => 3722m

• -Xmx should also be adjusted by the same amount 4512 + 3722 => 8234m

Java –Xms8234m –Xmx8234m –Xmn3722m –XX:MaxPermSize=96m

49

When Tuning Old Generation

Full GC too

long

Tune FullGC (start)

Measure worst case scenario FullGC Freq=> (Average increase in OldGen after MinorGC)

/MinorGC freq

Full GC freq. high

Increase Young Gen size

and/or adjust OldGen size yes

No

yes

Measure Full GC

duration

Measure Full GC Freq.

Adjust Young Gen size to: (TargetFreq/CurrentFreq) *

CurrentYoungGenSize

-Revisit Young Generation -Survivor Ratio calculation

-Calculate Object Promotion Rate from YoungGen to OldGen and SurvivorSpaceSize

-Or consider moving to CMS

50

CMS Collector

Survivor Space Size = -Xmn / (-XX:SurvivorRatio + 2 )

• Decrease Survivor Ratio causes an increase in Survivor Space Size

• Increase in Survivor Space Size causes Eden space to be reduced hence

• MinorGC frequency will increase

• More frequent MinorGC causes Objects to age quicker

• Use –XX:+PrintTenuringDistribution to measure how effectively objects age in survivor

spaces.

With CMS Collector increase –Xmx by 20-30%

• -XX:UseParNewGC is enabled by default when using:

• -XX:+UseConcMarkSweepGC

• Use –XX:CMSInitiatingOccupancyFraction=% (approx. 50%)

• Percentage should be adequate to accommodate about 1.5x the LiveData in OldGen

• Use –XX:CMSInitiatingOccupancyOnly

51

Best Practices

52

Enterprise Java Applications on vSphere Best Practices

Use the guidelines in the Enterprise Java Applications on VMware

– Best Practices Guide

(http://www.vmware.com/resources/techresources/1087 )

Set Memory Reservation

Set UseLargePages (set at JVM and OS level)

• In jRockit: -XXlargePages

• In HotSpot: -XX:+UseLargePages

• In IBM: -Xlp

53

Enterprise Java Applications on vSphere Best Practices

Set -Xmx equal to –Xms

• The cost of not setting –Xmx=-Xms, is that anytime the JVM has to resize a full

GC will be incurred. Which is not great for production response time

applications.

Set –Xmn (HotSpot) or –Xns (jRockit) to approx 33% of max heap.

• When using this fixed –Xmn option a resize of JVM will not cause the –Xmn to

change

PermGen

• In IBM it is dynamic and is the heap area, so you cant set it

• In HotSpot it is set by –XX:MaxPermSize approx < 1-6% of –Xmx, depends on

app

• In jRockit it is referred to as metadata, cannot be set directly, managed by JVM

54

Enterprise Java Applications on vSphere Best Practices

Leave Hyper-threading turned on, assume vCPU = 1.25 pCPU

ESX Host Clusters - When creating Cluster enable VMware HA and VMware

DRS

VMware DRS – enables vCenter Server to manage hosts as an aggregate

pool of resources.

Turn off Power Saving modes

55

Enterprise Java Applications on vSphere Best Practices

Use latest Nehalem or Shanghai based hardware

The VM-to-Host affinity: honor ISV Licensing, and anti-affinity ensure no 2

HA pairs on the same host.

VMs with a number of vCPUs equal to or less than the number of cores in

each NUMA node will be managed by the NUMA scheduler and will have

the best performance.

NUMA Node 1

Proc 1 Local

Memory Access

NUMA Node 1

Proc 1 Local

Memory Access

Remote Memory

Proc 1 Reaches

Proc 2 Memory

Or vice versa

GemFire VM

Running within

NUMA boundary

CPU

Cores

57

What are the practical and theoretical limits of JVM sizes

Windows 2008 (64-bit) is 2TB.

RHEL 5 is 1TB of RAM.

RHEL 6 is 2TB of RAM.

SUSE 11 is 16TB of RAM.

Practical NUMA localization limits depended on the amount of RAM

available in the server hardware you select, divided by the number of CPU

sockets on the server.

NUMA Local Memory = Total RAM on Server/Number of Processors

128 GB RAM

on Server

Each NUMA

Node has 128/2

64GB

4vCPU VM

Les than

64GB RAM

128 GB RAM

on Server

Each NUMA

Node has 128/2

64GB

2vCPU VMs

Less than

32GB RAM

on each VM

128 GB RAM

on Server

Each NUMA

Node has 128/2

64GB

4vCPU VMs

Less than

32GB RAM

on each VM

128 GB RAM

on Server

Each NUMA

Node has 128/2

64GB

2vCPU VMs

Less than

32GB RAM

on each VM ESX

Scheduler

128 GB RAM

on Server

Each NUMA

Node has 128/2

64GB

2vCPU VMs

Less than

32GB RAM

on each VM ESX

Scheduler

4vCPU VM

Split by ESX into

2 NUMA Clients

ESX4.1

128 GB RAM

on Server

Each NUMA

Node has 128/2

64GB

ESX

Scheduler

12vCPU VM

Can be done through

12 vCPU vSocket/vNUMA

in ESX5

64

Enterprise Java Applications on vSphere Best Practices

Use Load Balancers that integrate well with vSphere APIs

Load Balancer Algorithm Choice And VM Symmetry

Establish appropriate Thread ratios that Prevents bottlenecks

• (HTTP Threads:Java Threads:DB connections)

Googlebot or other web crawling robots somehow traverse to the

app, can cause performance degradation

• Best to divert the robot

65

Sizing Large JVMs

JVM Max Heap -Xmx

(29696m)

Perm Gen

Initial Heap

Guest OS Memory

-Xms (29696m)

Java Stack -Xss per thread (192k*100)

-XX:MaxPermSize (256m)

Other mem (=1484m)

500m used by OS

set mem Reservation to 31955m or set to Active

memory used by VM which could be lower.

JVM Memory

for GemFire

(31455m)

VM Memory

for GemFire (31955)

66

Sizing large JVMs

-XX:+UseNUMA in HotSpot JVM

• Only available Java 6 update 2

• Only available on –XX:+UseParallelOldGC and –XX:+UseParallelGC

• Only resort to NUMA tuning if it is a large JVM that spans multiple

memory/process nodes

• You can check to see N%L in esxtop if not a 100% then workload is not NUMA local

• Esxi5 NUMA optimizations are effective for majority of cases

• In Linux for singular JVMs use numacntl –interleave, for multiple JVMs you

could use –cpubind=<nodenumber> and –memnode=<nodenumber>

• we typically don‟t see the need to do this and the esx scheduler does a pretty good

job

67

Latency Sensitive Java Workloads on VMware – Best Practices

Disable NIC interrupt Coalescing on physical and virtual NIC

• Extremely helpful in reducing latency for latency-sensitive virtual machines

• disable virtual interrupt coalescing for VMXNET3

• It can lead to some performance penalties for other virtual machines on the ESXi

host, as well as higher CPU utilization to deal with the higher rate of interrupts from

the physical NIC.

• Latency Sensitive workloads best practices:

http://www.vmware.com/resources/techresources/10220

68

Most Common VM Size for Java workloads

2 vCPU VM

1 JVM (-Xmx 4096m)

Approx 5GB RAM Reservation

2 vCPU VM with 1 JVM, for tier-1 production workloads

Maintain this ratio as you scale out or scale-up, i.e. 1 JVM : 2vCPU

Scale out preferred over Scale-up, but both can work

You can diverge from this ratio for less critical workloads

69

High Availability

70 Confidential

High Availability Considerations of Enterprise Java Applications

DB ServersJava ApplicationsLoad balancers Web Servers

High Uptime, Scalable, and Dynamic Enterprise Java Applications

All tiers virtualized in VMware vSphere

71

High Availability Considerations of Enterprise Java Applications

Establish Zero-Downtime

Application Release Strategy

Phased approach to increasing uptime

Establish Dynamic Runtime

Mechanism for Handling Traffic

Bursts

Establish a Disaster Recovery Strategy

Phase 1

Phase 2

Phase 3

72

High Availability Phase 1 – Zero Downtime Releases

Step 1 (prior to release date)

Make a production replica of Production 1 called

Production 2

You can use Cloning/VM templates here

Step 2

Switch from Production 1 to Production 2 using

the load balancer mechanism

Step 3

Take a snapshot of Production 1 environment

VMs

Begin the application deployment/release

process using a master deployment script

Step 4

Synchronize the DBs between Production 1 and

Production 2 as more data may have been

introduced when Production 2 was live

Step 5

If per QA process the deployment is successful,

then switch at load balancer level to Production 1

If not successful, rollback to the last saved

snapshot in Step 3 and then switchback

Load Balancer

Production 1 Production 2

Step 2

Step 3

Step 1

Step 5

Step 4

73

High Availability Phase 1 – Zero Downtime Releases

Master Script in Apache Ant

<project name="MyProject" default="runMasterDeploymentScript"

basedir=".">

<target name =”runMasterDeploymentScript”

Depends=”takeSnapshotOfVM,

switchLoadBalancer,

deployDBChanges,

deployWARFile ”>

</target>

74

High Availability Phase 1 – Zero Downtime Releases

Master Script in Apache Ant

<target name="takeSnapshotOfVM">

<java

fork="true"

maxmemory="256m"

jvm="${JavaHome}"

classname="yourcompany.SnapshotManager" >

<arg value="-takeSnapshot"/> <arg value=”${vcenterURL}/>

<arg value=”${adminUsername}”/> <arg value=”${adminPwd}/>

<arg value=”${vmname}”/> <arg value=”${snapshotname}”/> <arg value=”${desc}”/>

<classpath>

<pathelement path="${VIJavaHome}/vijava2120100715.jar"/>

</classpath>

</java>

</target>

75

High Availability Phase 1 – Zero Downtime Releases

Master Script and VI Java API

public String takeSnapShotOfVM( String vcenterURL, String adminUsername, String adminPwd, String

vmname, String snapshotname, String desc)

{

ServiceInstance si = getServiceInstance(vcenterURL, adminUsername, adminPwd);

Folder rootFolder = si.getRootFolder();

String returnFlag = null;

try {

VirtualMachine vm = (VirtualMachine) new InventoryNavigator(

rootFolder).searchManagedEntity(VIRTUAL_MACHINE, vmname);

Task task = vm.createSnapshot_Task(snapshotname, desc, false, false);

if(task.waitForTask()==Task.SUCCESS)

{

returnFlag = Task.SUCCESS;

}else

{

returnFlag = "failure";

}

} catch (InterruptedException e) {

// other catch clauses omitted for brevity…

// other catch clauses go here…

}

return returnFlag;

}

76

High Availability Phase 1 – Zero Downtime Releases

VI Java API

• Getting started tutorial here:

• http://vijava.sourceforge.net/doc/getstarted/tutorial.htm

• Samples on creating:

• http://vijava.svn.sourceforge.net/viewvc/vijava/trunk/src/com/vmware/vim25/mo

/samples/

VM CloneVM,CreateVM,GetUpdates,MigrateVM,RemoveVmDisk,VMClone,V

MReconfig,VMSnapshot,VmNicOp,VmRename Cluster Create, Affinity rules , DRS vMotion history Storage AddDatastore,CopyFile,PrintStorageSystem,QueryVirtualDisk Network AddDVS,AddNIC,AddVirtualNic,AddVirtualSwitch,AddVirtualSwitchPortGro

up, FlipNetworkService, RemoveVirtualNic

77

High Availability Considerations of Enterprise Java Applications

Establish Zero-Downtime

Application Release Strategy

Phased approach to increasing uptime

Establish Dynamic Runtime

Mechanism for Handling Traffic

Bursts

Establish a Disaster Recovery Strategy

Phase 1

Phase 2

Phase 3

78

High Availability Phase 2 – Establish Dynamic Scalability

Vertical Scalability of VMs – CPU and Memory Hot Add

• Plan ahead and enable this feature

• The VM must be turned off to enable the Hot Plug feature. After enabling, Hot

Add CPU and Hot Add Memory can be done at runtime without shutting down

the VM so long as the guest OS supports it.

Ver

tica

l Sca

lab

ility

Add CPU/Memory/Storage

79

High Availability Phase 2 – Establish Dynamic Scalability

To Enable Hot Plug/Hot Add:

Go to VM > Edit Setting > Options tab

80

High Availability Phase 2 – Establish Dynamic Scalability

81

High Availability Phase 2 – Establish Dynamic Scalability

Horizontal Scalability

Add more Hosts

Horizontal Scalability

82

High Availability Phase 2 – Establish Dynamic Scalability

DRS

83

High Availability Phase 2 – Establish Dynamic Scalability

Load Balancer Integration

• Some load balancers have

an API that integrates with

VMware vCenter™ Server.

vCenter is aware if VMs

are added, so it can adjust

its pool configuration at runtime

• F5 for example has the

iControl API

• http://www.f5.com/solutions/applications/vmware/

vsphere/

• Graceful Server Shutdown,

• http://devcentral.f5.com/Tutorials/TechTips/tabi

d/63/articleType/ArticleView/articleId/254/iContro

l-Apps--04--Graceful-Server-Shutdown.aspx

Load Balancer

Newly added VMs automatically added to load

balancer

84

High Availability Considerations of Enterprise Java Applications

Establish Zero-Downtime

Application Release Strategy

Phased approach to increasing uptime

Establish Dynamic Runtime

Mechanism for Handling Traffic

Bursts

Establish a Disaster Recovery Strategy

Phase 1

Phase 2

Phase 3

85

High Availability Phase 3 – Disaster Recovery

Load Balancer Load Balancer

Global Load Balancer

Load Balancer Load Balancer

Global Load Balancer

SRM Array Replication

Datacenter Location-1 Datacenter Location-2

App clusterApp cluster

DRS Cluster DRS Cluster

86

Migration

87

Migration

Virtualize in Phases

• Phase 1 – Virtualize the Web Server Tier

• Phase 2 – Virtualize the Java Application Server Tier

• Phase 3 – Virtualize the DB Server Tier

• Phase 4 – Virtualize the Load Balancer Tier

DB ServersJava ApplicationsLoad balancers Web Servers

Enterprise Java Applications with all Tiers Virtualized

Phase 2 Phase 1 Phase 3 Phase 4

88

Migration

• On UNIX the UseLargePages flag is selected by default, so it must be turned

on in Linux because using large pages improves performance

• Smaller page sizes cause more TLB misses, hence increase page size to

reduce TLB miss and increase performance

• When you migrate your Java App to Linux use this setting:

• Set the –XX:+UseLargePages at the JVM level for Sun HotSpot

• Set the –XXlargePages at the JVM level for JRockit

• Set the –Xlp at the JVM level for IBM JVM

• Set the huge pages at OS level, refer to:

• http://www.vmware.com/resources/techresources/1087

• http://www.vmware.com/files/pdf/large_pg_performance.pdf

89

Migration

• Consider adding –XX:+AlwaysPreTouch when using large pages

• Enables touching all memory pages use by JVM heap on startup

• When GC copies objects to Survivor Spaces or promotes to OldGen, it needs new

pages and hence it may result in longer GC pauses as a result of trying zeroing and

committing memory pages. -XX:+AlwaysPreTouch avoids this cost, by incurring t

startup time.

90

Migration

When migrating from 32-bit to 64-bit JVM:

• the benefit is the ability to use larger Max Heap settings

• however, the same 32-bit application needs much more memory to run in 64-

bit due to its increased pointer address space requirements

• Performance can degrade in CPU cache efficiency, because less larger

references can fit into the CPU cache

• To mitigate: In Hotspot you can use –XX:+UseCompressedOops for up to 32GB, in

Java 6 update 18 this is enabled by default

• Use –d64 to specifically loads the 64 bit JVM instead of the default 32bit JVM

• Heap size between 2 and 32GB use –D64 and –XX:UseCompressedOops

• Heap size greater than 32GB: use –d64

91

Migration

• In IBM JVM, a similar flag exists it is called –Xcompressedrefs, up to 28G

• In Oracle jRockit it is –XXcompressedRefs up to 64G

92

Performance

93

Performance Perspective

• See the Performance of Enterprise Java Applications on VMware vSphere 4.1 and SpringSource tc Server at http://www.vmware.com/resources/techresources/10158 .

94

Performance Perspective

The 90th percentile response-time curves and CPU utilizations for

the 2 CPU native and virtualized cases.

Below 80% CPU utilization in the VM, the native and virtual

configurations have essentially identical performance, with only

minimal absolute differences in response-times.

80% Threshold

% CPU

R/T

95 Confidential

Performance Perspective

• The 90th percentile response-time curves for increasing load in the 4

CPU native and virtualized cases.

• The native and virtual configurations have essentially identical

performance across all loads, with only minimal differences in response-

times

80% Threshold

% CPU

R/T

96 Confidential

Performance Perspective

Shows the peak throughput for a single instance of Olio running on tc

Server, both natively and in a VM, with 1, 2, and 4 CPUs

97 Confidential

Performance Perspective

Number

of vCPUs

per VM

Number of

VMs

Per-VM

Maximum Heap

Size

Total Heap for 4vCPU Case

1 4 2GB 8GB 2 2 2.5GB 5GB 4 1 4GB 4GB

Lowest

CPU %

Best R/T

Best case

98 Confidential

Performance Perspective

2 VMs 2 vCPU each, 2.5G RAM on each VM –

showed best Throughput for the amount of memory used and R/T achieved

99 Confidential

Performance Perspective

Scaling of Peak Throughput

2 vCPU VMs shows best scalability when horizontally scaled.

100 Confidential

ESXi 5 Performance Enhancements

101 Confidential

ESXi 5 Enhancements

vMotion has three phases:

• Phase-1 (Guest Trace Phase):- traces are placed on the guest memory

pages to track any modifications by the guest during migration.

• Phase-2 (Precopy Phase): Iterative copy of changing memory from source to

destination

• Phase-3 (Switchover Phase):

• VM is momentarily quiesced on the source, the last set of memory

changes are copied to the target, and VM is resumed on target.

• During this phase Guest briefly pauses processing

• Duration is generally less than a second, however this could cause an

abrupt increase in latency

102 Confidential

ESXi 5 Enhancements

• Ability to use multiple network adapters for vMotion

• Latency-aware Metro vMotion that improves performance over long latency,

and increases round-trip tolerance limit for vMotion from 5msec to 10msec

• Stun During Page Send (SDPS) – ensure vMotion will not fail due to memory

convergence issues

• In ESXi 4.1 precopy process would fail to make forward progress due to

rapidly changing memory and hence fail the vMotion

• In ESXi 5 has the enhancement to slow down the memory changes in the

source VM, so that the precopy process can converge faster.

• Significant improvements to reduce memory tracing over head

103 Confidential

ESXi 5 Enhancements

In ESXi 5 vMotion is much faster than in ESX 4.1

• Performance : http://www.vmware.com/files/pdf/vmotion-perf-vsphere5.pdf

• SPECWeb2005, 12k Users (generated 6Gbps traffic), 4 vCPU VM with 12GB

RAM

• 37% improvement in vMotion duration

104 Confidential

ESXi 5 Enhancements

ESXi 5

ESXi 4.1

105 Confidential

ESXi 5 Enhancements

vMotion enhancements help more heavily loaded VMs

106 Confidential

Troubleshooting

107 Confidential

Enterprise Java Applications Troubleshooting Primer

Troubleshooting

• If you suspect vSphere is not configured optimally, you can open a

support ticket:

•http://www.vmware.com/support/contacts/file-sr.html

•Adhere to the troubleshooting steps outlined in the

http://communities.vmware.com/docs/DOC-10352

•Use the vm-support utility. You can execute this command at the

service console:

•vm-support –s

•this will collect necessary information about the symptoms so

that VMware can help diagnose the problem. It is best to run

this at the time the symptoms appear.

108 Confidential

Enterprise Java Applications Troubleshooting Primer

Troubleshooting

• References:

•Performance troubleshooting guide for ESX4.0

•http://communities.vmware.com/docs/DOC-10352

•The esxtop guide:

•http://communities.vmware.com/docs/DOC-11812

109 Confidential

Enterprise Java Applications Troubleshooting Primer - esxtop

Display Metric Threshold Description

CPU %RDY 10 Over provisioning of vCPU, excessive usage of vSMP or a limit (check %MLMTD) has been set. This %RDY value sum of all vCPUs %RDY for a VM. For example, if the max value of %RDY of 1vCPU is 100% and 4vCPU is 400%. If %RDY is 20 for 1 vCPU then this is problematic, as it means 1vCPU is waiting 20% of the time for VMkernel to schedule it.

CPU %CSTP 3 Excessive usage of vSMP. Decrease amount of vCPUs for this particular VM.

CPU %MLMTD 0 If larger than 0 the worlds being throttled. Possible cause is a limit on CPU

CPU %SWPWT 5 VM waiting on swapped pages to be read from disk. You may have over committed memory here.

110 Confidential

Enterprise Java Applications Troubleshooting Primer - esxtop

MEM MCTLSZ 1 If larger than 0 host is forcing VM to inflate balloon driver to reclaim memory as the host is overcommitted.

MEM SWCUR 1 If larger than 0 host has swapped memory pages in the past. You may have overcommitted.

MEM SWR/s 1 If larger than 0 host is actively reading from swap. This is caused by excessive memory over commitment

MEM SWW/s 1 If larger than 0 host is actively writing to swap. This is caused by excessive memory over commitment.

MEM N%L 80 If less than 80 VM experiences poor NUMA locality. If a VM has memory size greater than the amount of memory local to each processor, the ESX scheduler does not attempt to use NUMA optimizations for that VM.

111 Confidential

Enterprise Java Applications Troubleshooting Primer - esxtop

NETWORK %DRPTX 1 Dropped packages transmitted, hardware is

overworked due to high network utilization

NETWORK %DRPRX 1 Dropped packages received, hardware is overworked

due to high network utilization

112 Confidential

Enterprise Java Applications Troubleshooting Primer - esxtop

DISK GAVG 25 Look at DAVG and KAVG as GAVG = DAVG + KAVG

DISK DAVG 25 At this level you have Disk latency that is likely to be

caused by storage array

DISK KAVG 2 Disk latency caused by the VMkernel, high KAVG

usually means Queuing. Check QUED.

DISK QUED 1 Queue has maxed out. Possibly queue depth is set

too low. Check with array vendor for optimal queue

value.

DISK ABRTS/s 1 Aborts issued by VM because storage is not

responding. For Windows VMs this happens after 60

seconds default. Can be caused by path failure or

Storage Array Is not accepting IO.

DISK RESET/s 1 The number of commands resets per second

113 Confidential

Enterprise Java Applications Troubleshooting Primer

Java Troubleshooting

• Thread contention problems are notorious for causing scalability and

availability issues

•Memory Leak Issues are the next set of problems

•JDBC driver behavior is another

•Of course there are other slew of third party, or pure Java code problems

that can happen, these will need to be treated case by case basis.

114 Confidential

Enterprise Java Applications Troubleshooting Primer

Java Troubleshooting

• Thread Contention problems

•If you suspect that your Enterprise Java Application is suffering from

long pauses, or just general response time issues to the point where

the JVM needs to be restarted to resolve the issue.

•Then you need to inspect the Java Thread Dump.

•You can obtain a Java thread dump by CTRL-Break (in Windows OS)

or in Linux by issuing Kill -3 on the Java process ID. It is important

to take the thread dump right at the point where problematic

symptoms appear.

•If you are conducting a benchmark load test, it is advisable to take

thread dump at max peak load, and inspect the behavior of the

various application threads.

115 Confidential

Enterprise Java Applications Troubleshooting Primer

"[ACTIVE] ExecuteThread: '20' for queue: 'xyzAppServer.kernel.Default (self-tuning)'“

daemon prio=1 tid=0x082e1950 nid=0x2f9d runnable [0x7c96d000..0x7c96dec0]

at java.net.SocketInputStream.socketRead0(Native Method)

at java.net.SocketInputStream.read(SocketInputStream.java:129)

at xyzAppServer.jdbc.xyzDB.net8.XyzDBDataProvider.getArrayOfBytesFromSocket(Unknown Source)

at xyzAppServer.jdbc.xyzDB.net8.XyzDBDataProvider.readFirstPacketInBuffer(Unknown Source)

at xyzAppServer.jdbc.xyzDB.net8.XyzDBDataProvider.readPacket(Unknown Source)

at xyzAppServer.jdbc.xyzDB.net8.XyzDBDataProvider.receive(Unknown Source)

at xyzAppServer.jdbc.xyzDB.net8.XyzDBNet8NSPTDAPacket.sendRequest(Unknown Source)

at xyzAppServer.jdbc.xyzDB.XyzDBImplStatement.execute(Unknown Source)

at xyzAppServer.jdbc.base.BaseStatement.commonExecute(Unknown Source)

at xyzAppServer.jdbc.base.BaseStatement.executeInternal(Unknown Source)

at xyzAppServer.jdbc.base.BaseStatement.execute(Unknown Source)

- locked <0x8a5b7b38> (a xyzAppServer.jdbc.xyzDB.XyzDBConnection) at xyzAppServer.jdbc.wrapper.Statement.execute(Statement.java:400)

at jsp_servlet.__insert_test._jspService(__insert_test.java:108)

at xyzAppServer.servlet.jsp.JspBase.service(JspBase.java:34)

at xyzAppServer.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:226)

at xyzAppServer.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:124)

at xyzAppServer.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:283)

at xyzAppServer.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:175)

at xyzAppServer.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3370)

at xyzAppServer.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)

at xyzAppServer.security.service.SecurityManager.runAs(Unknown Source)

at xyzAppServer.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2117)

xyzAppServer.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2023)

at at xyzAppServer.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1359)

at xyzAppServer.work.ExecuteThread.execute(ExecuteThread.java:200)

at xyzAppServer.work.ExecuteThread.run(ExecuteThread.java:172)

Drill down to the “locked” thread

116 Confidential

Thanks you and are there any Questions?

Emad Benjamin,

ebenjamin@vmware.com

Book available here:

https://www.createspace.com/3632131