© 2010 VMware Inc. All rights reserved
Enterprise Java Applications on VMware
Technical Workshop
Emad Benjamin – VMware Staff Solutions Architect on Java and vFabric
2
About the speaker
I have been at VMware for the last 7 years, working on Java and
vSphere
20 years experience as a Software Engineer/Architect, with last 15
years focused on Java development
Open source contributions
Prior work with Cisco, Oracle, and Banking/Trading Systems.
Authored the Enterprise Java Applications Architecture on VMware
BCA1950
Production Proven Methods
of Running Enterprise Java
on vSphere.
Emad Benjamin, Staff Solutions Architect on Java and
vFabric, VMware, Inc.
4
Disclaimer
This session may contain product features that are
currently under development.
This session/overview of the new technology represents
no commitment from VMware to deliver these features in
any generally available product.
Features are subject to change, and must not be included in
contracts, purchase orders, or sales agreements of any kind.
Technical feasibility and market demand will affect final delivery.
Pricing and packaging for any new technologies or features
discussed or presented have not been determined.
5
Agenda
Conventional Enterprise Java Platforms
Enterprise Java Platform Architecture on VMware vSphere™
Design and Sizing
Best Practices
High Availability
Migration
Performance
Troubleshooting
Questions
6
Conventional Enterprise Java Platforms
7
Conventional Enterprise Java Platform
Enterprise Java applications are multitier
Multitier complexity leads to cascade impact when changes are
introduced
Multi-organizational challenges lead to increased cost
DB Servers Java Applications
Load Balancer Tier
Load Balancers Web Servers
IT Operations
Network Team
IT Operations
Server Team
IT Apps – Java
Dev Team
IT Ops & Apps
Dev Team
Organizational Key Stakeholder Departments
Web Server Tier Java App Tier DB Server Tier
8
Enterprise Java Platform Architecture
on VMware vSphere
9
Enterprise Java Applications Platform Architecture on vSphere
SHARED, ALWAYS-ONINFRASTRUCTURE
SHARED INFRASTRUCTURE SERVICES
Capacity On Demand High AvailabilityDynamic
APPLICATION SERVICES
DB ServersJava ApplicationsLoad balancers Web Servers
VMware vSphere
High Uptime, Scalable, and Dynamic Enterprise Java ApplicationsLoad Balancers as VMs
Web Servers
Java Application Servers
10
VMware vFabric Cloud Application Platform
Dynamic Load
Balancer
Performance
Management
Policy-driven
Automation
Elastic
App Server Cloud
Messaging
Global Data
Management
Application Infrastructure for the Cloud Generation
Virtual Datacenter Cloud Infrastructure and Management
tc Server ERS (Apache) Hyperic RabbitMQ GemFire
Frameworks & Tools
vFabric Platform Services
Rich Web Integration Batch Data Access Social /Cloud Tool Suite
11
Choice of frameworks
.js
12
Data
Services
Other
Services
Msg
Services
Choice of application services
.js
13
Private Clouds
Public Clouds
Micro Clouds
Choice of clouds
.js
Data
Services
Other
Services
Msg
Services
.COM
…
14
Cloud Foundry – key use cases
New applications
Dev-test-trial SaaS extensibility
App modernization
15
Cloud Foundry – key audiences
Grassroots Developers “No one between you and your application”
Friction-free way to develop, test and deploy
Open ecosystem of frameworks and services
Use it your way – open source, micro clouds, choice of public clouds
Self-service access to cloud-scale application resources
SaaS ISVs
“Cloudify your apps”
Elastic programmability and extensibility to your SaaS service
High productivity frameworks and application services
16
Cloud Foundry – under the covers
Router
Infrastructure
Health Manager
DEA: execution environment Services: core, ecosystem,
etc.
user apps Cloud Controller user apps
vmc client STS plugin browser (user app
access)
17
Cloud Foundry – key audiences
Create app, update app, control app
vmc push [appname] [--path] [--url] [--instances N] [--mem] [--no-start]
vmc update <appname> [--path PATH]
vmc stop <appname>
vmc start <appname>
vmc target [url]
Update app settings, get app information
vmc mem <appname> [memsize]
vmc map <appname> <url>
vmc instances <appname> <num | delta>
vmc {crashes, crashlogs, logs} <appname>
vmc files <appname> [path]
18
Cloud Foundry – key audiences
Deal with services, users, and information
vmc create-service <service> [--name servicename] [--bind appname]
vmc bind-service <servicename> <appname>
vmc unbind-service <servicename> <appname>
vmc delete-service <servicename>
vmc user, vmc passwd, vmc login, vmc logout, vmc add-user
vmc services, vmc apps, vmc info
19
Design and Sizing
20
Design and Sizing of Enterprise Java Applications on vSphere
Step 1 –
Establish Load profile
From production logs/monitoring reports measure:
Concurrent UsersRequests Per SecondPeak Response TimeAverage Response Time
Establish your response time SLA
Step 2
Establish Benchmark
Iterate through Benchmark test until you are satisfied with the Load profile metrics and your intended SLAafter each benchmark iteration you may have to adjust the Application Configuration Adjust the vSphereenvironment to scale out/up in order to achieve your desired number of VMs, number of vCPU and RAM configurations
Step 3 –
Size Production Env.
The size of the production environment would have been established in Step2, hence either you roll out the environment from Step-2 or build a new one based on the numbers established
Step 2 – Establish Benchmark
DETERMINE HOW MANY VMs Establish Horizontal Scalability
Scale Out Test How many VMs do you need to meet your Response Time SLAs without reaching 70%-80% saturation of CPU? Establish your Horizontal scalability Factor before bottleneck appear in your application
Scale Out Test
Building Block VM Building Block VM
SLA OK?
Test complete
Investigate bottlnecked layer Network, Storage, Application
Configuration, & vSphere
If scale out bottlenecked
layer is removed, iterate scale out
test
If building block app/VM config problem, adjust
& iterate No
Building Block VM
ESTABLISH BUILDING BLOCK VM Establish Vertical scalability
Scale Up Test Establish how many JVMs on a VM? Establish how large a VM would be in terms of vCPU and memory
Scal
e U
p T
est
Building Block VM
22
Design and Sizing HotSpot JVMs on vSphere
JVM Max Heap -Xmx
JVM Memory
Perm Gen
Initial Heap
Guest OS Memory
VM Memory
-Xms
Java Stack
-Xss per thread
-XX:MaxPermSize
Other mem
Direct
native
memory
Non
Direct
Memory
Virtual
Address
space
23
Design and Sizing of HotSpot and jRockit JVMs on vSphere
Guest OS Memory approx 1G (depends on OS/other processes)
Perm Size is an area additional to the –Xmx (Max Heap) value and
is not GC-ed because it contains class-level information.
“other mem” is additional mem required for NIO buffers, JIT code
cache, classloaders, Socket Buffers (receive/send), JNI, GC internal
info
If you have multiple JVMs (N JVMs) on a VM then:
• VM Memory = Guest OS memory + N * JVM Memory
VM Memory = Guest OS Memory + JVM Memory
JVM Memory = JVM Max Heap (-Xmx value) + JVM Perm Size (-XX:MaxPermSize) +
NumberOfConcurrentThreads * (-Xss) + “other Mem”
24
Design and Sizing of IBM JVM on vSphere
JVM Max Heap -Xmx
JVM Memory
Perm Gen
Initial Heap
Guest OS Memory
VM Memory
-Xms
Java Stack
-Xss per thread
-XX:MaxPermSize
Other mem
Perm Gen
25
Design and Sizing of jRockit JVM on vSphere
JVM Max Heap -Xmx
JVM Memory
Perm Gen
Initial Heap
Guest OS Memory
VM Memory
-Xms
Java Stack
-Xss per thread
Other mem
Metadata Dynamically managed by JVM
26
Sizing Example
JVM Max Heap -Xmx
(4096m)
JVM Memory (4588m) Perm Gen
Initial Heap
Guest OS Memory
VM Memory (5088m)
-Xms (4096m)
Java Stack -Xss per thread (192k*100)
-XX:MaxPermSize (256m)
Other mem (=217m)
500m used by OS
set mem Reservation to 5088m or set to Active
memory used by VM which could be lower.
27
Sizing The Java Heap
JVM Max Heap -Xmx
(4096m)
Eden Space
Survivor Space 2
Old Generation
Survivor Space 1
Slower
Full GC
Quick
Minor GC YoungGen -Xmn
(1350m)
OldGen 27460m
28
But how did we establish –Xmx is 4096MB?
One answer is that we know from testing the application
Alternately: we started the Java application without –Xmx or –Xms
being specified, and then monitor the heap usage during a load
test.
• You are determining the actual active data used in OldGen after a fullGC
• Observe whether you are getting OldGen or PermGen OutOfMemory Errors
Above approach helps you determine live memory “LiveData”
• LiveData is equivalent to OldGen space occupancy after FullGC
• As a general rule size –Xmx (the Heap) => 3 to 4 times the “LiveData”
• PermGen is 1%-6% of –Xmx, or 1.5 times the PermGen after FullGC
• YoungGen (-Xmn) approx 33% of –Xmx, or 1.5 times LiveData after FullGC
29
But how did we establish –Xmx is 4096MB?
2011-10-15T19:59:09.895-0600:
[Full GC
[PSYoungGen: 958,034k-> 0k (1,254,400k)]
[ParOldGen: 1,494,122k-> 1,032,888 k(2,398,095k)]
[ 2,452,156k -> 1,032,888k (3,669,995k)]
[PSPermGen: 113,365k-> 113,365 k(163,840k)],
0.874769secs]
Young Gen
(-Xmn)
Old Gen
(-Xmx “ –” -Xmn)
Perm Gen size
(-XX:MaxPermSize)
Young Gen
before GC
Old Gen
before GC
Perm Gen size
before GC
Full GC
duration
Old Gen
after FullGC
active data “LiveData”
1GB
30
But how did we establish –Xmx is 4096MB?
Note: numbers before “->” are before GC, and numbers after “->”
are after GC numbers.
Since Active Data was approx 1GB, and applying the rule of Heap (-
Xmx) should be 3 to 4 times of Active data size, hence chose Heap
of 4096MB in our sizing example.
Then we applied all of the other relative sizing rules to establish the
other memory segments that we already covered.
Then we used an amount of 5088MB as Memory Reservation for
the VM when all the various memory segments were added up.
31
Sizing Example for a 4096MB Heap and 5GB Reserved VM Mem
JVM Max Heap -Xmx
(4096m)
JVM Memory (4588m) Perm Gen
Initial Heap
Guest OS Memory
VM Memory (5088m)
-Xms (4096m)
Java Stack -Xss per thread (192k*100)
-XX:MaxPermSize (256m)
Other mem (=217m)
500m used by OS
set mem Reservation to 5088m or set to Active
memory used by VM which could be lower.
32
Design and Sizing:
Understanding ESX Memory
Management
33
Why did we use Memory Reservation?
Memory Reservation
• We used an amount of 5088MB as Memory Reservation for the VM when all
the various memory segments were added up.
• This is the physical ESX Host memory guaranteed to be available to the VM
up-on startup such that memory over commitment is avoided
• Hence ESX memory management techniques such as Ballooning and
Swapping avoided in order to preserve performance
34
ESX Memory Management
Guest virtual memory is mapped to Guest Physical is in turn mapped to
ESX Physical Memory
35
ESX Memory Management
The below shows an over committed memory situation, only 4GB
physical is available but 6GB has been allocated to the VMs
ESX uses Transparent Page Sharing (TPS), Ballooning, and host
swapping, in order to support memory reclamation in an overcommitted
situation
36
Understand ESX Memory Management
ESX uses Transparent Page Sharing (TPS) – when multiple VMs are
running it is highly likely that common memory segments exit, for
example Guest OS used is common amongst the VMs, and hence can
be TPS-ed
37
ESX Memory Management
TPS scans both VM and Host memory for common memory segments
to cache
Scan rate controlled by Mem.ShareScanTime, Mem.SharerateMax, and
Mem.ShareScanGHz
38
ESX Memory Management
Ballooning – makes the Guest OS aware that ESX Host memory is short
Due to VMs isolation, the Guest OS is not aware that it is in a VM and is
not aware of the state of other VMs, or the ESX Host memory situation
Balloon driver is loaded into Guest OS, communicates with ESX via
private channel, and ESX will instruct it to inflate (allocate Guest OS
Physical Pages) by an amount. An amount that it needs to reclaim.
39
ESX Memory Management
ESX Host Swapping – this is a last resort, and things are quite bad at
this stage. If TPS and Ballooning didn‟t work, then ESX will swap VM
memory to swap file.
40
Use ESX to Share Memory Between VMs Running Java
Elastic Memory for Java (EM4J)
Description Benefits
• Higher consolidation through memory over-commit for
Java workloads
• Protect Java applications against workload spikes
without wasting memory • Lets you give Java more memory than it needs under
normal load. Memory your application doesn‟t need is
returned to ESX but remains available in case it is
needed to avoid Java out-of-memory errors
• Use virtualization to
overcome the
limitation of Java‟s
static heap
• New „memory balloon‟
runs inside JVM to
capture unused
memory and return it
to ESX
• Memory returned to
JVM when it is under
pressure
41
Design and Sizing:
Undersating GC
42
Which GC?
ESX doesn’t care which GC you select, because of the degree of
independence of Java to OS and OS to Hypervisor
43
Which GC?
Either you tune for Throughput or Latency, one at the cost of the
other
Start with Throughput collector, if latency is not met then move to
Concurrent collector
• Throughput Collector -XX:+UseParallelOldGC enables both multithreaded
young generation and old generation collector
• If -XX:+UseParallelOldGC not available then you can use –XX:UseParallelGC,
but it only allows for multithreaded young generation.
• When using –XX:UseParallelGC the old generation garbage collector used
with this option is the serial collector which is single threaded.
• If you find latency is an issue, then you can move to the CMS collector.
44
When tuning GC use GC logs
To investigate use GCLogging
49.09:
[GC
[PSYoungGen: 958,034k-> 0k (1,254,400k)]
[ 2,452,156k -> 1,032,888k (3,669,995k), 0.108 secs]
[Times: user=1.83 sys=0.01, real=0.11 secs0.874769secs]
-XX:+PrintGCTimeStamps –XX:+PrintGCDetails –Xloggc:<filename>
45
What to measure when tuning GC?
General approach to investigating latency
• Determine Minor GC duration
• Determine Minor GC Frequency
• Determine worst Full GC duration
• Determine worst Full GC frequency
Minor GC measurements drive Young Generation refinements
Full GC measurements drive Old Generation refinements
The decision to switch to –XX:+UseConcMarkSweepGC
• If throughput collector's worst case FullGC duration/frequency compared with
app latency
46
When Tuning Young Generation
Minor GC too
long
Tune MinorGC
(start)
Reduce Young Gen size
Minor GC
freq. high
Increase Young Gen size
yes
No
yes
Measure Minor GC duration
Reduce OldGen size by the same amount as you reduced
YoungGen
Measure Minor GC
Freq.
Adjust Young Gen size to: (TargetFreq/CurrentFreq) *
CurrentYoungGenSize -Also adjust OldGen by same amount
47
Tuning Example Adjust Duration of Minor GC
Java –Xms4512m –Xmx4512m –Xmn1489m –XX:MaxPermSize=96m
Avg GC Minor GC Freq = 2 seconds
Avg Minor GC Duration = 0.05 seconds, ie 50msec
App latency requirements is 40mseconds
• Measured latency 54msec, hence duration too long MUST REDUCE
YoungGen
• Reduce Young Gen by 10% and heap by the same amount
Java –Xms4060m –Xmx4060m –Xmn1340m –XX:MaxPermSize=96m
48
Tuning Example Adjust Frequency of Minor GC
Average Minor GC Freq = 2 seconds
But supposed instead of pause time, the app cares more about
frequency of GC
• Frequency Requirements is 5seconds
• Increase –Xmn (YoungGen) to reduce GC frequency to 5seconds, 5/2=>2.5
• Original –Xmn is 1489, adjusted value is 2.5*1489 => 3722m
• -Xmx should also be adjusted by the same amount 4512 + 3722 => 8234m
Java –Xms8234m –Xmx8234m –Xmn3722m –XX:MaxPermSize=96m
49
When Tuning Old Generation
Full GC too
long
Tune FullGC (start)
Measure worst case scenario FullGC Freq=> (Average increase in OldGen after MinorGC)
/MinorGC freq
Full GC freq. high
Increase Young Gen size
and/or adjust OldGen size yes
No
yes
Measure Full GC
duration
Measure Full GC Freq.
Adjust Young Gen size to: (TargetFreq/CurrentFreq) *
CurrentYoungGenSize
-Revisit Young Generation -Survivor Ratio calculation
-Calculate Object Promotion Rate from YoungGen to OldGen and SurvivorSpaceSize
-Or consider moving to CMS
50
CMS Collector
Survivor Space Size = -Xmn / (-XX:SurvivorRatio + 2 )
• Decrease Survivor Ratio causes an increase in Survivor Space Size
• Increase in Survivor Space Size causes Eden space to be reduced hence
• MinorGC frequency will increase
• More frequent MinorGC causes Objects to age quicker
• Use –XX:+PrintTenuringDistribution to measure how effectively objects age in survivor
spaces.
With CMS Collector increase –Xmx by 20-30%
• -XX:UseParNewGC is enabled by default when using:
• -XX:+UseConcMarkSweepGC
• Use –XX:CMSInitiatingOccupancyFraction=% (approx. 50%)
• Percentage should be adequate to accommodate about 1.5x the LiveData in OldGen
• Use –XX:CMSInitiatingOccupancyOnly
51
Best Practices
52
Enterprise Java Applications on vSphere Best Practices
Use the guidelines in the Enterprise Java Applications on VMware
– Best Practices Guide
(http://www.vmware.com/resources/techresources/1087 )
Set Memory Reservation
Set UseLargePages (set at JVM and OS level)
• In jRockit: -XXlargePages
• In HotSpot: -XX:+UseLargePages
• In IBM: -Xlp
53
Enterprise Java Applications on vSphere Best Practices
Set -Xmx equal to –Xms
• The cost of not setting –Xmx=-Xms, is that anytime the JVM has to resize a full
GC will be incurred. Which is not great for production response time
applications.
Set –Xmn (HotSpot) or –Xns (jRockit) to approx 33% of max heap.
• When using this fixed –Xmn option a resize of JVM will not cause the –Xmn to
change
PermGen
• In IBM it is dynamic and is the heap area, so you cant set it
• In HotSpot it is set by –XX:MaxPermSize approx < 1-6% of –Xmx, depends on
app
• In jRockit it is referred to as metadata, cannot be set directly, managed by JVM
54
Enterprise Java Applications on vSphere Best Practices
Leave Hyper-threading turned on, assume vCPU = 1.25 pCPU
ESX Host Clusters - When creating Cluster enable VMware HA and VMware
DRS
VMware DRS – enables vCenter Server to manage hosts as an aggregate
pool of resources.
Turn off Power Saving modes
55
Enterprise Java Applications on vSphere Best Practices
Use latest Nehalem or Shanghai based hardware
The VM-to-Host affinity: honor ISV Licensing, and anti-affinity ensure no 2
HA pairs on the same host.
VMs with a number of vCPUs equal to or less than the number of cores in
each NUMA node will be managed by the NUMA scheduler and will have
the best performance.
NUMA Node 1
Proc 1 Local
Memory Access
NUMA Node 1
Proc 1 Local
Memory Access
Remote Memory
Proc 1 Reaches
Proc 2 Memory
Or vice versa
GemFire VM
Running within
NUMA boundary
CPU
Cores
57
What are the practical and theoretical limits of JVM sizes
Windows 2008 (64-bit) is 2TB.
RHEL 5 is 1TB of RAM.
RHEL 6 is 2TB of RAM.
SUSE 11 is 16TB of RAM.
Practical NUMA localization limits depended on the amount of RAM
available in the server hardware you select, divided by the number of CPU
sockets on the server.
NUMA Local Memory = Total RAM on Server/Number of Processors
128 GB RAM
on Server
Each NUMA
Node has 128/2
64GB
4vCPU VM
Les than
64GB RAM
128 GB RAM
on Server
Each NUMA
Node has 128/2
64GB
2vCPU VMs
Less than
32GB RAM
on each VM
128 GB RAM
on Server
Each NUMA
Node has 128/2
64GB
4vCPU VMs
Less than
32GB RAM
on each VM
128 GB RAM
on Server
Each NUMA
Node has 128/2
64GB
2vCPU VMs
Less than
32GB RAM
on each VM ESX
Scheduler
128 GB RAM
on Server
Each NUMA
Node has 128/2
64GB
2vCPU VMs
Less than
32GB RAM
on each VM ESX
Scheduler
4vCPU VM
Split by ESX into
2 NUMA Clients
ESX4.1
128 GB RAM
on Server
Each NUMA
Node has 128/2
64GB
ESX
Scheduler
12vCPU VM
Can be done through
12 vCPU vSocket/vNUMA
in ESX5
64
Enterprise Java Applications on vSphere Best Practices
Use Load Balancers that integrate well with vSphere APIs
Load Balancer Algorithm Choice And VM Symmetry
Establish appropriate Thread ratios that Prevents bottlenecks
• (HTTP Threads:Java Threads:DB connections)
Googlebot or other web crawling robots somehow traverse to the
app, can cause performance degradation
• Best to divert the robot
65
Sizing Large JVMs
JVM Max Heap -Xmx
(29696m)
Perm Gen
Initial Heap
Guest OS Memory
-Xms (29696m)
Java Stack -Xss per thread (192k*100)
-XX:MaxPermSize (256m)
Other mem (=1484m)
500m used by OS
set mem Reservation to 31955m or set to Active
memory used by VM which could be lower.
JVM Memory
for GemFire
(31455m)
VM Memory
for GemFire (31955)
66
Sizing large JVMs
-XX:+UseNUMA in HotSpot JVM
• Only available Java 6 update 2
• Only available on –XX:+UseParallelOldGC and –XX:+UseParallelGC
• Only resort to NUMA tuning if it is a large JVM that spans multiple
memory/process nodes
• You can check to see N%L in esxtop if not a 100% then workload is not NUMA local
• Esxi5 NUMA optimizations are effective for majority of cases
• In Linux for singular JVMs use numacntl –interleave, for multiple JVMs you
could use –cpubind=<nodenumber> and –memnode=<nodenumber>
• we typically don‟t see the need to do this and the esx scheduler does a pretty good
job
67
Latency Sensitive Java Workloads on VMware – Best Practices
Disable NIC interrupt Coalescing on physical and virtual NIC
• Extremely helpful in reducing latency for latency-sensitive virtual machines
• disable virtual interrupt coalescing for VMXNET3
• It can lead to some performance penalties for other virtual machines on the ESXi
host, as well as higher CPU utilization to deal with the higher rate of interrupts from
the physical NIC.
• Latency Sensitive workloads best practices:
http://www.vmware.com/resources/techresources/10220
68
Most Common VM Size for Java workloads
2 vCPU VM
1 JVM (-Xmx 4096m)
Approx 5GB RAM Reservation
2 vCPU VM with 1 JVM, for tier-1 production workloads
Maintain this ratio as you scale out or scale-up, i.e. 1 JVM : 2vCPU
Scale out preferred over Scale-up, but both can work
You can diverge from this ratio for less critical workloads
69
High Availability
70 Confidential
High Availability Considerations of Enterprise Java Applications
DB ServersJava ApplicationsLoad balancers Web Servers
High Uptime, Scalable, and Dynamic Enterprise Java Applications
All tiers virtualized in VMware vSphere
71
High Availability Considerations of Enterprise Java Applications
Establish Zero-Downtime
Application Release Strategy
Phased approach to increasing uptime
Establish Dynamic Runtime
Mechanism for Handling Traffic
Bursts
Establish a Disaster Recovery Strategy
Phase 1
Phase 2
Phase 3
72
High Availability Phase 1 – Zero Downtime Releases
Step 1 (prior to release date)
Make a production replica of Production 1 called
Production 2
You can use Cloning/VM templates here
Step 2
Switch from Production 1 to Production 2 using
the load balancer mechanism
Step 3
Take a snapshot of Production 1 environment
VMs
Begin the application deployment/release
process using a master deployment script
Step 4
Synchronize the DBs between Production 1 and
Production 2 as more data may have been
introduced when Production 2 was live
Step 5
If per QA process the deployment is successful,
then switch at load balancer level to Production 1
If not successful, rollback to the last saved
snapshot in Step 3 and then switchback
Load Balancer
Production 1 Production 2
Step 2
Step 3
Step 1
Step 5
Step 4
73
High Availability Phase 1 – Zero Downtime Releases
Master Script in Apache Ant
<project name="MyProject" default="runMasterDeploymentScript"
basedir=".">
<target name =”runMasterDeploymentScript”
Depends=”takeSnapshotOfVM,
switchLoadBalancer,
deployDBChanges,
deployWARFile ”>
</target>
74
High Availability Phase 1 – Zero Downtime Releases
Master Script in Apache Ant
<target name="takeSnapshotOfVM">
<java
fork="true"
maxmemory="256m"
jvm="${JavaHome}"
classname="yourcompany.SnapshotManager" >
<arg value="-takeSnapshot"/> <arg value=”${vcenterURL}/>
<arg value=”${adminUsername}”/> <arg value=”${adminPwd}/>
<arg value=”${vmname}”/> <arg value=”${snapshotname}”/> <arg value=”${desc}”/>
<classpath>
<pathelement path="${VIJavaHome}/vijava2120100715.jar"/>
</classpath>
</java>
</target>
75
High Availability Phase 1 – Zero Downtime Releases
Master Script and VI Java API
public String takeSnapShotOfVM( String vcenterURL, String adminUsername, String adminPwd, String
vmname, String snapshotname, String desc)
{
ServiceInstance si = getServiceInstance(vcenterURL, adminUsername, adminPwd);
Folder rootFolder = si.getRootFolder();
String returnFlag = null;
try {
VirtualMachine vm = (VirtualMachine) new InventoryNavigator(
rootFolder).searchManagedEntity(VIRTUAL_MACHINE, vmname);
Task task = vm.createSnapshot_Task(snapshotname, desc, false, false);
if(task.waitForTask()==Task.SUCCESS)
{
returnFlag = Task.SUCCESS;
}else
{
returnFlag = "failure";
}
} catch (InterruptedException e) {
// other catch clauses omitted for brevity…
// other catch clauses go here…
}
return returnFlag;
}
76
High Availability Phase 1 – Zero Downtime Releases
VI Java API
• Getting started tutorial here:
• http://vijava.sourceforge.net/doc/getstarted/tutorial.htm
• Samples on creating:
• http://vijava.svn.sourceforge.net/viewvc/vijava/trunk/src/com/vmware/vim25/mo
/samples/
VM CloneVM,CreateVM,GetUpdates,MigrateVM,RemoveVmDisk,VMClone,V
MReconfig,VMSnapshot,VmNicOp,VmRename Cluster Create, Affinity rules , DRS vMotion history Storage AddDatastore,CopyFile,PrintStorageSystem,QueryVirtualDisk Network AddDVS,AddNIC,AddVirtualNic,AddVirtualSwitch,AddVirtualSwitchPortGro
up, FlipNetworkService, RemoveVirtualNic
77
High Availability Considerations of Enterprise Java Applications
Establish Zero-Downtime
Application Release Strategy
Phased approach to increasing uptime
Establish Dynamic Runtime
Mechanism for Handling Traffic
Bursts
Establish a Disaster Recovery Strategy
Phase 1
Phase 2
Phase 3
78
High Availability Phase 2 – Establish Dynamic Scalability
Vertical Scalability of VMs – CPU and Memory Hot Add
• Plan ahead and enable this feature
• The VM must be turned off to enable the Hot Plug feature. After enabling, Hot
Add CPU and Hot Add Memory can be done at runtime without shutting down
the VM so long as the guest OS supports it.
Ver
tica
l Sca
lab
ility
Add CPU/Memory/Storage
79
High Availability Phase 2 – Establish Dynamic Scalability
To Enable Hot Plug/Hot Add:
Go to VM > Edit Setting > Options tab
80
High Availability Phase 2 – Establish Dynamic Scalability
81
High Availability Phase 2 – Establish Dynamic Scalability
Horizontal Scalability
Add more Hosts
Horizontal Scalability
82
High Availability Phase 2 – Establish Dynamic Scalability
DRS
83
High Availability Phase 2 – Establish Dynamic Scalability
Load Balancer Integration
• Some load balancers have
an API that integrates with
VMware vCenter™ Server.
vCenter is aware if VMs
are added, so it can adjust
its pool configuration at runtime
• F5 for example has the
iControl API
• http://www.f5.com/solutions/applications/vmware/
vsphere/
• Graceful Server Shutdown,
• http://devcentral.f5.com/Tutorials/TechTips/tabi
d/63/articleType/ArticleView/articleId/254/iContro
l-Apps--04--Graceful-Server-Shutdown.aspx
Load Balancer
Newly added VMs automatically added to load
balancer
84
High Availability Considerations of Enterprise Java Applications
Establish Zero-Downtime
Application Release Strategy
Phased approach to increasing uptime
Establish Dynamic Runtime
Mechanism for Handling Traffic
Bursts
Establish a Disaster Recovery Strategy
Phase 1
Phase 2
Phase 3
85
High Availability Phase 3 – Disaster Recovery
Load Balancer Load Balancer
Global Load Balancer
Load Balancer Load Balancer
Global Load Balancer
SRM Array Replication
Datacenter Location-1 Datacenter Location-2
App clusterApp cluster
DRS Cluster DRS Cluster
86
Migration
87
Migration
Virtualize in Phases
• Phase 1 – Virtualize the Web Server Tier
• Phase 2 – Virtualize the Java Application Server Tier
• Phase 3 – Virtualize the DB Server Tier
• Phase 4 – Virtualize the Load Balancer Tier
DB ServersJava ApplicationsLoad balancers Web Servers
Enterprise Java Applications with all Tiers Virtualized
Phase 2 Phase 1 Phase 3 Phase 4
88
Migration
• On UNIX the UseLargePages flag is selected by default, so it must be turned
on in Linux because using large pages improves performance
• Smaller page sizes cause more TLB misses, hence increase page size to
reduce TLB miss and increase performance
• When you migrate your Java App to Linux use this setting:
• Set the –XX:+UseLargePages at the JVM level for Sun HotSpot
• Set the –XXlargePages at the JVM level for JRockit
• Set the –Xlp at the JVM level for IBM JVM
• Set the huge pages at OS level, refer to:
• http://www.vmware.com/resources/techresources/1087
• http://www.vmware.com/files/pdf/large_pg_performance.pdf
89
Migration
• Consider adding –XX:+AlwaysPreTouch when using large pages
• Enables touching all memory pages use by JVM heap on startup
• When GC copies objects to Survivor Spaces or promotes to OldGen, it needs new
pages and hence it may result in longer GC pauses as a result of trying zeroing and
committing memory pages. -XX:+AlwaysPreTouch avoids this cost, by incurring t
startup time.
90
Migration
When migrating from 32-bit to 64-bit JVM:
• the benefit is the ability to use larger Max Heap settings
• however, the same 32-bit application needs much more memory to run in 64-
bit due to its increased pointer address space requirements
• Performance can degrade in CPU cache efficiency, because less larger
references can fit into the CPU cache
• To mitigate: In Hotspot you can use –XX:+UseCompressedOops for up to 32GB, in
Java 6 update 18 this is enabled by default
• Use –d64 to specifically loads the 64 bit JVM instead of the default 32bit JVM
• Heap size between 2 and 32GB use –D64 and –XX:UseCompressedOops
• Heap size greater than 32GB: use –d64
91
Migration
• In IBM JVM, a similar flag exists it is called –Xcompressedrefs, up to 28G
• In Oracle jRockit it is –XXcompressedRefs up to 64G
92
Performance
93
Performance Perspective
• See the Performance of Enterprise Java Applications on VMware vSphere 4.1 and SpringSource tc Server at http://www.vmware.com/resources/techresources/10158 .
94
Performance Perspective
The 90th percentile response-time curves and CPU utilizations for
the 2 CPU native and virtualized cases.
Below 80% CPU utilization in the VM, the native and virtual
configurations have essentially identical performance, with only
minimal absolute differences in response-times.
80% Threshold
% CPU
R/T
95 Confidential
Performance Perspective
• The 90th percentile response-time curves for increasing load in the 4
CPU native and virtualized cases.
• The native and virtual configurations have essentially identical
performance across all loads, with only minimal differences in response-
times
80% Threshold
% CPU
R/T
96 Confidential
Performance Perspective
Shows the peak throughput for a single instance of Olio running on tc
Server, both natively and in a VM, with 1, 2, and 4 CPUs
97 Confidential
Performance Perspective
Number
of vCPUs
per VM
Number of
VMs
Per-VM
Maximum Heap
Size
Total Heap for 4vCPU Case
1 4 2GB 8GB 2 2 2.5GB 5GB 4 1 4GB 4GB
Lowest
CPU %
Best R/T
Best case
98 Confidential
Performance Perspective
2 VMs 2 vCPU each, 2.5G RAM on each VM –
showed best Throughput for the amount of memory used and R/T achieved
99 Confidential
Performance Perspective
Scaling of Peak Throughput
2 vCPU VMs shows best scalability when horizontally scaled.
100 Confidential
ESXi 5 Performance Enhancements
101 Confidential
ESXi 5 Enhancements
vMotion has three phases:
• Phase-1 (Guest Trace Phase):- traces are placed on the guest memory
pages to track any modifications by the guest during migration.
• Phase-2 (Precopy Phase): Iterative copy of changing memory from source to
destination
• Phase-3 (Switchover Phase):
• VM is momentarily quiesced on the source, the last set of memory
changes are copied to the target, and VM is resumed on target.
• During this phase Guest briefly pauses processing
• Duration is generally less than a second, however this could cause an
abrupt increase in latency
102 Confidential
ESXi 5 Enhancements
• Ability to use multiple network adapters for vMotion
• Latency-aware Metro vMotion that improves performance over long latency,
and increases round-trip tolerance limit for vMotion from 5msec to 10msec
• Stun During Page Send (SDPS) – ensure vMotion will not fail due to memory
convergence issues
• In ESXi 4.1 precopy process would fail to make forward progress due to
rapidly changing memory and hence fail the vMotion
• In ESXi 5 has the enhancement to slow down the memory changes in the
source VM, so that the precopy process can converge faster.
• Significant improvements to reduce memory tracing over head
103 Confidential
ESXi 5 Enhancements
In ESXi 5 vMotion is much faster than in ESX 4.1
• Performance : http://www.vmware.com/files/pdf/vmotion-perf-vsphere5.pdf
• SPECWeb2005, 12k Users (generated 6Gbps traffic), 4 vCPU VM with 12GB
RAM
• 37% improvement in vMotion duration
104 Confidential
ESXi 5 Enhancements
ESXi 5
ESXi 4.1
105 Confidential
ESXi 5 Enhancements
vMotion enhancements help more heavily loaded VMs
106 Confidential
Troubleshooting
107 Confidential
Enterprise Java Applications Troubleshooting Primer
Troubleshooting
• If you suspect vSphere is not configured optimally, you can open a
support ticket:
•http://www.vmware.com/support/contacts/file-sr.html
•Adhere to the troubleshooting steps outlined in the
http://communities.vmware.com/docs/DOC-10352
•Use the vm-support utility. You can execute this command at the
service console:
•vm-support –s
•this will collect necessary information about the symptoms so
that VMware can help diagnose the problem. It is best to run
this at the time the symptoms appear.
108 Confidential
Enterprise Java Applications Troubleshooting Primer
Troubleshooting
• References:
•Performance troubleshooting guide for ESX4.0
•http://communities.vmware.com/docs/DOC-10352
•The esxtop guide:
•http://communities.vmware.com/docs/DOC-11812
109 Confidential
Enterprise Java Applications Troubleshooting Primer - esxtop
Display Metric Threshold Description
CPU %RDY 10 Over provisioning of vCPU, excessive usage of vSMP or a limit (check %MLMTD) has been set. This %RDY value sum of all vCPUs %RDY for a VM. For example, if the max value of %RDY of 1vCPU is 100% and 4vCPU is 400%. If %RDY is 20 for 1 vCPU then this is problematic, as it means 1vCPU is waiting 20% of the time for VMkernel to schedule it.
CPU %CSTP 3 Excessive usage of vSMP. Decrease amount of vCPUs for this particular VM.
CPU %MLMTD 0 If larger than 0 the worlds being throttled. Possible cause is a limit on CPU
CPU %SWPWT 5 VM waiting on swapped pages to be read from disk. You may have over committed memory here.
110 Confidential
Enterprise Java Applications Troubleshooting Primer - esxtop
MEM MCTLSZ 1 If larger than 0 host is forcing VM to inflate balloon driver to reclaim memory as the host is overcommitted.
MEM SWCUR 1 If larger than 0 host has swapped memory pages in the past. You may have overcommitted.
MEM SWR/s 1 If larger than 0 host is actively reading from swap. This is caused by excessive memory over commitment
MEM SWW/s 1 If larger than 0 host is actively writing to swap. This is caused by excessive memory over commitment.
MEM N%L 80 If less than 80 VM experiences poor NUMA locality. If a VM has memory size greater than the amount of memory local to each processor, the ESX scheduler does not attempt to use NUMA optimizations for that VM.
111 Confidential
Enterprise Java Applications Troubleshooting Primer - esxtop
NETWORK %DRPTX 1 Dropped packages transmitted, hardware is
overworked due to high network utilization
NETWORK %DRPRX 1 Dropped packages received, hardware is overworked
due to high network utilization
112 Confidential
Enterprise Java Applications Troubleshooting Primer - esxtop
DISK GAVG 25 Look at DAVG and KAVG as GAVG = DAVG + KAVG
DISK DAVG 25 At this level you have Disk latency that is likely to be
caused by storage array
DISK KAVG 2 Disk latency caused by the VMkernel, high KAVG
usually means Queuing. Check QUED.
DISK QUED 1 Queue has maxed out. Possibly queue depth is set
too low. Check with array vendor for optimal queue
value.
DISK ABRTS/s 1 Aborts issued by VM because storage is not
responding. For Windows VMs this happens after 60
seconds default. Can be caused by path failure or
Storage Array Is not accepting IO.
DISK RESET/s 1 The number of commands resets per second
113 Confidential
Enterprise Java Applications Troubleshooting Primer
Java Troubleshooting
• Thread contention problems are notorious for causing scalability and
availability issues
•Memory Leak Issues are the next set of problems
•JDBC driver behavior is another
•Of course there are other slew of third party, or pure Java code problems
that can happen, these will need to be treated case by case basis.
114 Confidential
Enterprise Java Applications Troubleshooting Primer
Java Troubleshooting
• Thread Contention problems
•If you suspect that your Enterprise Java Application is suffering from
long pauses, or just general response time issues to the point where
the JVM needs to be restarted to resolve the issue.
•Then you need to inspect the Java Thread Dump.
•You can obtain a Java thread dump by CTRL-Break (in Windows OS)
or in Linux by issuing Kill -3 on the Java process ID. It is important
to take the thread dump right at the point where problematic
symptoms appear.
•If you are conducting a benchmark load test, it is advisable to take
thread dump at max peak load, and inspect the behavior of the
various application threads.
115 Confidential
Enterprise Java Applications Troubleshooting Primer
"[ACTIVE] ExecuteThread: '20' for queue: 'xyzAppServer.kernel.Default (self-tuning)'“
daemon prio=1 tid=0x082e1950 nid=0x2f9d runnable [0x7c96d000..0x7c96dec0]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at xyzAppServer.jdbc.xyzDB.net8.XyzDBDataProvider.getArrayOfBytesFromSocket(Unknown Source)
at xyzAppServer.jdbc.xyzDB.net8.XyzDBDataProvider.readFirstPacketInBuffer(Unknown Source)
at xyzAppServer.jdbc.xyzDB.net8.XyzDBDataProvider.readPacket(Unknown Source)
at xyzAppServer.jdbc.xyzDB.net8.XyzDBDataProvider.receive(Unknown Source)
at xyzAppServer.jdbc.xyzDB.net8.XyzDBNet8NSPTDAPacket.sendRequest(Unknown Source)
at xyzAppServer.jdbc.xyzDB.XyzDBImplStatement.execute(Unknown Source)
at xyzAppServer.jdbc.base.BaseStatement.commonExecute(Unknown Source)
at xyzAppServer.jdbc.base.BaseStatement.executeInternal(Unknown Source)
at xyzAppServer.jdbc.base.BaseStatement.execute(Unknown Source)
- locked <0x8a5b7b38> (a xyzAppServer.jdbc.xyzDB.XyzDBConnection) at xyzAppServer.jdbc.wrapper.Statement.execute(Statement.java:400)
at jsp_servlet.__insert_test._jspService(__insert_test.java:108)
at xyzAppServer.servlet.jsp.JspBase.service(JspBase.java:34)
at xyzAppServer.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:226)
at xyzAppServer.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:124)
at xyzAppServer.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:283)
at xyzAppServer.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:175)
at xyzAppServer.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3370)
at xyzAppServer.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at xyzAppServer.security.service.SecurityManager.runAs(Unknown Source)
at xyzAppServer.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2117)
xyzAppServer.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2023)
at at xyzAppServer.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1359)
at xyzAppServer.work.ExecuteThread.execute(ExecuteThread.java:200)
at xyzAppServer.work.ExecuteThread.run(ExecuteThread.java:172)
Drill down to the “locked” thread
116 Confidential
Thanks you and are there any Questions?
Emad Benjamin,
Book available here:
https://www.createspace.com/3632131
Top Related