Evidence based performance tuning of

enterprise Java applicationsenterprise Java applications

By Jeroen Borgersjborgers@xebia.com

Evidence based performance tuning of

enterprise Java applications

By Jeroen Borgersjborgers@xebia.com

enterprise Java applications

Bad performance can lead to complete failure

3

Bad performance: hard to fix

• Call center agent took 20 minutes to work through five screens with customer– Customer un-friendly

– High cost

• Speedup needed before rush of number porting

4

• Speedup needed before rush of number porting– Major app redesign was needed

• Result– very slow system when up, mostly system down

– damage to company image

– loss of thousands of customers and $100,000,000

– company crash

Goal

Provide an answer to the question:

How can I speedup my enterprise Java app?

5

Who is Jeroen Borgers?

• sr. consultant with Xebia

• helps customers on enterprise Java performance issues

• instructs Java performance tuning courses• instructs Java performance tuning courses

• doing Java since 1996

• various roles in several industries

– developer, architect, team lead, quality officer, auditor, performance tester and tuner

www.xebia.com

Who is Xebia?

• international IT organization

– NL, FR, IN

• specialized in enterprise Java

• consultancy and projects• consultancy and projects

• 5 years old

• ±100 employees and growing

• share knowledge

• focus on quality

www.xebia.com

Agenda

• Big picture architecture vs. implementation

• How to achieve bad performance

• Evidence based tuning for high speed

• Tools overview

8

• Tools overview

• Largest Dutch web shop case

• Conclusions

Enterprise Java performance big picture

• Java is not slow anymore

• Importance how you do it: algorithms

• Typical enterprise app: most time spent in calls out of Java process– Remote calls

9

– Remote calls

– Database access

for (int i = 0; i < size; i++) {

order = orderDao.getOrder(i);

if (order.isDelivered()) { DB

>~10 ms~0,1µs

factor 100.000 !

Architecture enables performance and scalability

• Key choices– Application distribution

– Persistent data access approach: O-R mapping

– Data conversion: XML, encodings

– Presentation tier technologies

10

– Presentation tier technologies

– Use of proprietary features of db, app server

– Data caching

• It is all about:– Remote calls

– Database access• More general: data conversion between representations

Architecture enables, implementation makes it happen

• Minimize remote calls and other I/O

• Speed-up data conversion

11

• Code optimizations involving remote calls and data conversion

?

Architecture enables, implementation makes it happen

• Minimize remote calls and other I/O– Direct JDBC: use of batch updates

– Hibernate: use of query caching

• Speed-up data conversion– XML usage: use JiBX instead of JAXB

12

– XML usage: use JiBX instead of JAXB

– URL encoding: use commons codec instead of java.net

• Code optimizations involving remote calls and data conversion– Algorithms, collections

– Threading and synchronization

Architecture enables performance & scalability

• The architecture

13

Architecture enables performance & scalability

• The architecture is scalable

14

?

Architecture enables performance & scalability

• The architecture is scalable

15

• Add horse power: more load with same speed

Architecture enables performance & scalability

• How one implements the architecture makes a difference

16

• But the requirement of 120 km/h will never be met!

• Major redesign needed

Architecture enables performance & scalability

• A more suitable architecture for 120 km/h:

17

Architecture enables performance & scalability

• More demanding requirements can be met with the right implementation

• Bugatti EB Veyron 16.4

18

16.4

• Top 407 km/h

• 0-300 in 14 s.

How to achieve bad performance

• Many JavaEE apps have hard to solve performance problems

• How to achieve this?

– Architecture not suitable for requirements

19

– Architecture not suitable for requirements

– Ignore the problems too long

– Don’t follow best practices

– Optimize counterproductively

Architecture not suitable

• Architect: “Most important is a pure, flexible, scalable architecture: we distribute our application components and loosely couple with XML and web services.”

20

• Nonsense!

– Those who pay the bill care about efficient working, not purity;

– Speed is traded for scalability: remote calls and data conversion can be killing;

– How is flexibility gained?

Ignore problems too long

• Project leader: “First we get the functionality right, then we start worrying about performance.”

• Performance is often ignored until the final test, a week before production, or later!

• To meet requirements you may need a major re-

21

• To meet requirements you may need a major re-design: costly

• Solved by:– Verify the architecture in a POC: test a vertical slice

– Test performance continuously

How to achieve bad performance

• How to achieve this?

– Architecture not suitable for requirements

– Ignore the problems too long

– Don’t follow best practices

22

– Don’t follow best practices

– Optimize counterproductively

Don’t follow best practices

• Best practices are like:

– Use Map instead of List for lookups

– Use I/O buffering: InputStreamBuffer

– Use SQL batch updates

23

– Use SQL batch updates

– Use db table indexes

– Use StringBuffer for String concatenation?

?

Don’t follow best practices

• Best practices are like:

– Use Map instead of List for lookups

– Use I/O buffering: InputStreamBuffer

– Use SQL batch updates

24

– Use SQL batch updates

– Use db table indexes

– Use StringBuffer for String concatenation?

• Be aware: a best practice may not work in your situation

Best practices can be premature optimizations with trade-offs

• Developer: “String performs badly, we use StringBuffer for concatenation everywhere.”

• Many books: “use StringBuffer fortoString() { return “[“ + getName() + “]”; }

• faster istoString() {

StringBuffer buf = new StringBuffer(“[“);

buf.append(getName()).append(“]”);

25

buf.append(getName()).append(“]”);

return buf.toString();

} ”• However: same bytecode!• Maintainability and developer time is traded for an assumed, non-

existing performance gain• “Premature optimization is the root of all evil” - Hoare and Knuth

Optimize counterproductively

• Updating state remotely by sending only delta’s like RemoveAttribute, AddAttribute– many small objects with much serialization and

communication overhead

• Object and thread pooling depending on JVM

26

• Object and thread pooling depending on JVM

• Optimizations are very time and environment dependent

• So: don’t trust your knowledge or intuition, but measure that your optimization solves the problem!

Bad performance

• Architecture not suitable for requirements

• Ignore the problems too long

• Don’t follow best practices

• Optimize counterproductively

27

• Optimize counterproductively

How to speedup?

Evidence based tuning for high speed

• Don’t make assumptions but measure• Have quantified requirements, prove they are met or not• Prove the architecture performs in a POC• Continuously test performance during development

– to get a first impression– for quick feedback on changes

28

– for quick feedback on changes

• Test representatively in a dedicated environment– with production-like load– on a production-like environment– with production-like data

• Fix problems when found• Prove the effect of each optimization

• Monitor the performance of your app in production

Evidence based tuning cycle

Tools for measuring

• OS profilers to see resource usage: CPU, memory, I/O, threading • Database analyzers to see query behavior in the database• Network analyzers to see network traffic• App server monitors to see app server resource usage: heap,

connection pool, stmt cache,..• J2EE analyzers to see J2EE behavior under load• Instrumentation monitors to see timing at the instrumented

30

• Instrumentation monitors to see timing at the instrumented locations

• Java profilers to see detailed CPU and memory behavior in JVM• Continuous test tools to quickly get timing feedback of code

changes• Load test tools to generate usage load• Web site analyses tools to optimize page resource usage

Largest Dutch web shop case

www.xebia.com ?

Case: speeding up a web shop

32

Wehkamp TRC architecture

web server: ASP/.NetTuxedo

DB

33

TRC-appsrv: Java

DB

Services here

DB

Speedup results at Wehkamp

• Several services optimized to meet requirements

• Most expensive: voegKlantToe (addCustomer); verwerkenNieuweOrder(processNewOrder): from 7 s. to 1.5 s.

• 45% db-queries, 45% tux-calls, 10% java code

• Top optimizations:

34

• Top optimizations:– Reduce number of db queries

– Optimize most expensive db queries

– Reduce number of Tux/mainframe calls

– Optimize most expensive Tux/mainframe calls

• App orchestrates: many Java code changes needed for this

Tools used

• Load test tool: Apache JMeter• App level monitor: JAMon API• History and trends reporter: JARep• Continuous test tool: cruisecontrol+maven+JMeter• Java profiler: Quest JProbe

35

• Java profiler: Quest JProbe• App server monitor: Tivoli Performance Monitor• J2EE analyzer: Quest PerformaSure• Database analyzer: Quest Toad, Oracle Statspack• Unix OS profiler: vmstat, nmon• Network analyzer: Wireshark

?

JMeter for generating load and black box testing

36

JMeter mail to stakeholders

37

JAMon for monitoring statistics

added

service name

38

Instrumenting with JAMon API

39

JAMon API call

JARep reporting for history

40

41

42

43

JVM / app server node 4

JVM / app server node 3

deployment of JARep

your app

jamon.warjamonadmin.jsp

ShowJamonCounters.jsp DataFetcher

DataPersister

every 5 minutes

JVM / app server node 3

JVM / app server node 2

JVM / app server node 1

www.xebia.com

jamonapi.jar

JVM / off-line box

RDBMS

JaReporter.jsp

PerformaSure for analyses under load

45

Analysis tools compared

• Profiler JProbe– Pro: High detail: code line level

– Con: High overhead, cannot deal with load

• JAMon API + JARep– Pro: Low overhead < 1%

46

– Pro: Low overhead < 1%

– Pro: Enables comparing over time, see trends

– Con: No call tree, cannot drill down

– Con: Limited to built-in measure points

• PerformaSure– Pro: reasonable overhead, slick UI, measures all on JVM

– Con: not cheap, configuration, wait time

Conclusions & recommendations

• Architecture and implementation both determine performance

• Consider whole application life cycle: requirements, POC, continuous testing, representative testing, monitoring

47

representative testing, monitoring

• Biggest gains are in– Remote calls and other I/O

– Database access or other data conversions

• Tuning based on evidence is key– measure and prove by using the right tools

– JMeter, JAMon and JARep can help

“Meten is weten” (measuring means knowing)

48

References

• http://www.xebia.com• http://blog.xebia.com• http://podcast.xebia.com• http://www.javaperformancetuning.com• http://sourceforge.net/projects/jarep

49

• http://sourceforge.net/projects/jarep• http://www.jamonapi.com• http://jakarta.apache.org/jmeter/• http://www.quest.com/performance_management

Q & A

50