Frank Houweling

AMIS preview Oracle Open World 2014

Real-World Load Testing of ADF Fusion Applications

Frank Houweling

• Frank Houweling

• Senior Java / ADF specialist

• Focus on performance diagnosis and performance management

• Lead developer of the ADF Performance Monitor

3

Agenda

• Why is testing important

• Intro Apache JMeter and Oracle Load Testing (OLT)

• Demo

• Metrics analysis

• Top 3 scalability problems and solutions

4

Why is load testing important - 1

5

Why is load testing important - 2

• Is very useful to test scalability and SLA

– Avoid frustrated end-users

– Avoid lost revenue

Challenges Load Testing ADF Application

• Trying out / testing the most optimal:

– ApplicationModule pooling parameter settings

– JVM heap size & garbage collection strategy

– Datasource configuration (number of database connections)

– Number of WebLogic nodes

– Infrastructure/hardware usage & licenses

7

You need metrics to be in control and know what's going on

8

Introducing Apache JMeter

• Open source desktop application, designed to load test functional behavior and measure performance.

• open source software

• 100% pure Java application

• not easy to configure

• but free

Introducing Oracle Application Testing Suite (OATS)

• Powerful, integrated solution for ensuring

application quality.

Oracle Functional Testing

Oracle Test Manager

Oracle Load Testing

It is Oracle’s solution for testing Oracle Applications and the standard for testing at Oracle For example Fusion/ADF, EBS, Siebel, JDE, FMW, e.g.

10

OpenScript Automates functional and load testing

• OpenScript is used for both creation of functional and load test scripts using record and playback

• Accelerators (templates) for Fusion/ADF, webservices, eBS, JDE, e.g.

• Integrated Eclipse IDE for script development

11

OpenScript Concepts

12

Installing OATS

• Current newest version: 12.4.0.1.0

• Download from OTN

13

Agenda

• Why is testing important

• Intro Apache JMeter and Oracle Load Testing (OLT)

• Demos (NOTE: without sound, play in Firefox or IE)

– Record a load test scenario (1.20 min)

– Execute a load test (3 min)

• Metrics analysis

• Top 3 scalability problems and solutions

14

Analysis load test metrics: Default graphs in OLT

• Default graphs

– Active Virtual Users • number of virtual users currently running

– Virtual Users with Errors • number of virtual users with errors

– Transactions Per Second • number of times the virtual user played back the script per second

– Pages Per Second • number of pages returned by the server per second

– Hits Per Second • number of resource requests to the server per second

– Kilobytes Per Second • number of kilobytes transferred between the server and browser client per second

15

Analysis load test metrics: Determine the weakest links

• Collect relevant load test metrics

– Number of virtual users

– Average response times

– ApplicationModule pooling activations and passivations

– JVM heap and garbage collections

– Database performance

– Infrastructure metrics

• Bring the metrics together in time

• Analyze results and determine weak links

16

Analysis load test metrics - 1 example: CPU seems to be a bottleneck

throughput virtual users

CPU usage

• imported CPU metrics from WLS during load test – the CPU seems to be the bottleneck

17

Analysis load test metrics - 2 Monitor in Database EM Control during load test

Example: High CPU Wait in Database EM Control

18

Agenda

• Why is testing important

• Intro Apache JMeter and Oracle Load Testing (OLT)

• Demo

• Metrics analysis

• Top 3 scalability problems and solutions

Common scalability problems - 1 Too frequent ‘expensive’ ApplicationModule

passivations & activations

Recommendations

• Don’t use the AM pooling default parameter settings; calculate how many AMs you will

need during peek times

– Increase maxavailablesize and recyclethreshold to this value • jbo.ampool.maxavailablesize = jbo.recyclethreshold

– Set minavailablesize to 80% of maxavailablesize • jbo.ampool.minavailablesize = 80 % of jbo.ampool.maxavailablesize

Result:

• Avoids long running, ‘expensive’ passivations and activations of AMs under normal load

• Best practice: test it during a load test !

20 Monitor ApplicationModule Pooling behavior; activations and passivations

• What is the average, total, min and max execution time of activations and passivations?

21

What is the root cause of long passivations?

22

What is the root cause of long activations?

23 Common scalability problems - 2 Too little JVM Heap size, long running JVM

garbage collections

• Avoid too frequent long running JVM garbage collections that freeze all requests

• Set ( -Xms and -Xmx) as large as possible within available physical memory

• Generational parallel garbage collection strategy is recommended to maximize

throughput: -Xgc:genpar (JRockit)

• Consider more Weblogic nodes if you have many end-users

• Best practice: test it during a load test !

24

Common scalability problems - 3 Not enough database connections

Solution: increase Maximum Capacity of datasource in WLS console

25 Monitor nr of database connections in WLS console during load test

• Monitor the number of database connections in the Weblogic Console

• set jbo.doconnectionpooling=true and jbo.txn.disconnect_level=1

26 Measure number of database connections in database during load test

(logged in as SYS or SYSDBA)

27

Tuning Challenges

• Requires expertise and is time consuming

– Should be done in time (not 1 week before production)

– Relevant scenario’s should be tested

– All load testing tools take time to become familiar

• Analysis

– What type of operations is the ADF application or database spending most time on ?

– Which resources is the WLS server or database bottlenecked on?

– What is causing these bottlenecks?

– What can be done to resolve them ?

28

Learn More

• Oracle Application Testing Suite on OTN

– Download

– Tutorials

– Demos

– Training

• Free alternative (but more configuration):

Apache Jmeter

• Fusion Developer Guide on ApplicationModule pooling (11.1.2)

• ADF Performance Monitor