Oracle troubleshooting

8/16/2019 Oracle troubleshooting

1/128

Oracle WebLogic Server11g : Diagnostics andTroubleshooting

Activity Guide

D61523GC20

Edition 2.0

March 2011

D72555


2/128

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Disclaimer

This document contains proprietary information and is protected by copyright and other intellectual property laws. You may copy andprint this document solely for your own use in an Oracle training course. The document may not be modified or altered in any way.Except where your use constitutes "fair use" under copyright law, you may not use, share, download, upload, copy, print, display,perform, reproduce, publish, license, post, transmit, or distribute this document in whole or in part without the express authorizationof Oracle.

The information contained in this document is subject to change without notice. If you find any problems in the document, pleasereport them in writing to: Oracle University, 500 Oracle Parkway, Redwood Shores, California 94065 USA. This document is notwarranted to be error-free.

Restricted Rights Notice

If this documentation is delivered to the United States Government or anyone using the documentation on behalf of the UnitedStates Government, the following notice is applicable:

U.S. GOVERNMENT RIGHTSThe U.S. Government’s rights to use, modify, reproduce, release, perform, display, or disclose these training materials are restrictedby the terms of the applicable Oracle license agreement and/or the applicable U.S. Government contract.

Trademark Notice

Oracle and Java are registered trademarks of Oracle Corporation and/or i ts affiliates. Other names may be trademarks of their

respective owners.

Author

Bill Bell

Technical Contributors and Reviewers

Will Lyons, TJ Palazzolo, Serge Moiseev

This book was published using: Oracle Tutor


3/128


Oracle WebLogic Server 11g: Diagnostics and Troubleshooting Table of Contents

i

Table of Contents

Practices for Lesson 1 .....................................................................................................................................1-1

Practices for Lesson 1 ....................................................................................................................................1-2



Practice 2-1: Connecting to the Classroom Grid ............................................................................................2-3

Practice 2-2: Developing a Custom Monitoring Script ....................................................................................2-5

Solution Instructions .......................................................................................................................................2-9



Practice 3-1: Using Guardian to Evaluate a Domain ......................................................................................3-3




Practice 4-1: Harvesting Diagnostic Metrics ...................................................................................................4-3


Practice 4-2: Monitoring Diagnostic Metrics ...................................................................................................4-9



Practice 5-1: Configuring and Monitoring Diagnostic Events .........................................................................5-3


Practice 5-2: Tracing a Client Request ...........................................................................................................5-7




Practice 6-1: Troubleshooting a Running JVM ...............................................................................................6-3

Practice 6-2: Troubleshooting Applications on JRockit ..................................................................................6-7



Practice 7-1: Investigating Classpath Problems .............................................................................................7-3



Practice 8-1: Investigating Server Problems ..................................................................................................8-3




Practice 9-1: Investigating JDBC Problems ...................................................................................................9-3


Practices for Lesson 10 ...................................................................................................................................10-1

Practices for Lesson 10 ..................................................................................................................................10-2

Practice 10-1: Investigating JMS Problems ....................................................................................................10-3



4/128


Oracle WebLogic Server 11g: Diagnostics and Troubleshooting Table of Contents

ii



Practice 11-1: Investigating SSL Problems ....................................................................................................11-3


Practice 11-2: Investigating Security Realm Problems ...................................................................................11-10




Practice 12-1: Investigating Node Manager Problems ...................................................................................12-3




Practice 13-1: Investigating Proxy Problems ..................................................................................................13-3


Practice 13-2: Investigating Cluster Replication Problems .............................................................................13-8


5/128


Practices for Lesson 1

Chapter 1 - Page 1


Chapter 1


6/128



Chapter 1 - Page 2


Practices Overview

There are no practices for this lesson.


7/128



Chapter 2 - Page 1


Chapter 2


8/128



Chapter 2 - Page 2


Practices Overview

In the practices for this lesson, you connect to your practice machine for the first time. You thenwrite a WLST script to monitor a running server.

Naming Conventions

The instructions for the course practices use variable names to refer to frequently usedlocations on your file system. Some of these are also defined as Linux environment variables.Other variable names will be introduced in later practices as needed.

Variable Path

/u01/app/oracle/Middleware/11.1.1

/wlserver_10.3

/home/oracle/wls11g_trouble

/labs/LabXX_YY ,where XX_YY is the current practice number

/work


9/128



Chapter 2 - Page 3

Practice 2-1: Connecting to the Classroom Grid

Duration: 10 minutes

Skills Learned

At the end of this practice, you should be able to:

• Identity your classroom virtual machine (VM) instance

• Connect to a classroom virtual machine by using an NX client• Establish an NX session

Overview

Oracle University employs a Grid VM infrastructure to allow students to work with complex labenvironments from anywhere in the world. You will be assigned a specific Linux VM instance onthis Grid for you to perform the subsequent course practices. In this practice, you useNoMachine, an NX client, to interact with this remote VM as if it were running on your localmachine.

Instructions

1. Obtain your VM hostname, username, and password from your instructor. Grid hosts aretypically of the form egNNNN .us.oracle.com.

2. From the Windows Start menu, select All Programs > NX Client for Windows > NXConnection Wizard.

3. Click Next.

4. Enter the following values (NNNN is your assigned VM instance number):

Field Value

Session egNNNN

Host egNNNN .us.oracle.com

Click Next.

5. Select the GNOME option and click Next.

6. Click Finish.

The NX Login window appears.


10/128



Chapter 2 - Page 4

7. Enter your assigned Username and Password. Confirm that your new session, egNNNN,is selected. Click Login.

8. Confirm that your VM Linux desktop is displayed.

Additional Notes: − When you leave your remove session, click the “X” in the top-right corner of the

window. You do not need to Terminate your remote session, instead you maychoose Disconnect, which leaves programs running on the remote system. Youmay leave programs running on the remote system for the duration of the course.

− After a period of inactivity, your remote session may lock. If this occurs, you mustsimply enter your Password to resume the session.

Solution Instructions

No solution exists for this practice.


11/128



Chapter 2 - Page 5

Practice 2-2: Developing a Custom Monitoring Script


Skills Learned


• Navigate the WLS runtime MBean hierarchy

•

Retrieve and display MBean attributes• Use loop and condition Jython constructs

Overview

The Oracle Medical Records Java EE application, abbreviated MedRec, provides patients withthe ability to view current and previous medical examinations and prescriptions over the Web.MedRec, like all Java EE applications, is dependent on various server resources. Theseresources include JDBC data sources, JMS queues and topics, and shared libraries.

In this practice, you develop a custom WLST script that periodically monitors the health of theMedRec application, its host server, and its supporting resources.

The MedRec domain infrastructure used in this practice is depicted in the following diagram:

Instructions

1. Initialize the MedRec server infrastructure.

a. Launch the Lab Framework command shell by executing the

/bin/prompt.sh file.

Tip: Remember that you can refer to the Naming Conventions section earlier in thisguide for the definition of variables such as .

b. Change the current directory to .

c. Execute the following:

ant setup_exercise

Note: When the Ant script completes its work you will see the message “BUILD

SUCCESSFUL” in the Lab Framework command shell.

Important: You must log in to each WebLogic Server as it comes up (see below).

The Lab Framework creates files and folders in your directory. The

framework:

− Starts the Oracle database and loads the MedRec schema

− Creates a skeleton MedRec domain from a template


12/128



Chapter 2 - Page 6

− Starts the administration server and a managed server (be sure to log in as

weblogic/Welcome1 as each server boots)

− Creates a JDBC data source

− Creates a JMS server and module

− Creates a work manager

− Deploys the shared libraries

− Deploys the MedRec application

− Copies a test client to , which you will use in a later practice.

2. Connect to the server runtime MBean hierarchy.

a. Create a new folder: /scripts.

b. Launch an editor and create a new file:

/scripts/monitorMedRec.py.

c. Use the connect() WLST command to connect to the MedRecSvr1 managed server(port 7021).

Tip: Remember, the three arguments you will be using with connect() are an admin-level username, password, and server URL. Each argument must be delimited (forexample in single quotes). The protocol of the URL, if omitted, defaults to t3, which iswhat you want.

d. Use the serverRuntime() command to switch to the server’s runtime hierarchy.

3. Print server, application, and data source MBean attributes.

a. Create a new while loop whose contents iterate indefinitely, as shown below.

while 1:

# Loop contents here

Tip: Remember that in Jython, you use whitespace (a tab, for example) to denote thecontents of a loop or condition.

b. Within the loop, locate this server’s ThreadPoolRuntime MBean and retrieve itsthroughput attribute. For example:

threadPool =

getMBean('/ThreadPoolRuntime/ThreadPoolRuntime')

throughput = threadPool.getThroughput()

c. Similarly, retrieve the status attribute of the following application MBean:

/ApplicationRuntimes/medrec/ComponentRuntimes/MedRecSvr1_/medrec

Tip: This MBean represents the Web module with the context path /medrec, found inthe enterprise application medrec.ear.

d. Get the value of the openSessionsCurrentCount attribute from the same MBean.

e. Retrieve the following MBean:/JDBCServiceRuntime/MedRecSvr1/JDBCDataSourceRuntimeMBeans/MedRecGlobalDataSourceXA

f. Get the following attributes from this MBean:

state

numAvailable

activeConnectionsCurrentCount


13/128



Chapter 2 - Page 7

g. Print all the collected attributes.

Tip: You can convert numeric values to text by using either of the following techniques:

print 'Text ', num

print 'Text ' + str(num)

Tip: Optionally, you can format floating point values such as throughput by usingcode like the following:

print '%.1f'%(throughput)

h. Add the following import statement to the top of the file:

import time as jythontime

i. Add a 1 second pause within the while loop:

jythontime.sleep(1)

Tip: Alternatively, you can call the equivalent operation in the Java language:Thread.sleep(1000).

j. Save your script.

4. Monitor the server without any load.

a. Locate your Lab Framework prompt or start a new one. Navigate to/scripts.

b. Use WLST to run your new script:

java weblogic.WLST monitorMedRec.py

c. Confirm that the current status of the application is DEPLOYED and the state of thedata source is Running. Correct any errors.

Tip: If you need it, a completed script can be found at /solution.

Tip: If you must restart your servers at any point in the course, use thestartWebLogic.sh and startServer1.sh scripts found in your domain.

d. Leave the script running.

5. Monitor the server under load.

a. Launch a second Lab Framework prompt by using /bin/prompt.sh.

b. Navigate to /client.

c. Execute the runclient.sh script.

Note: You may ignore any warnings in the MedRecSvr1 window about logging.

d. Return to your monitoring script output. Confirm that the number of active sessions hasincreased by 1. The data source active connections may have increased by 1 as well.

After the client completes its execution, if the number of active connections went to 1, itshould return to 0.

Note: You may not see the active database connections go up. It is very possible that

a database connection is retrieved, used, and returned within the 1 second sleep time.e. Execute the runclients.sh script.

This script simulates 10 concurrent users.


14/128



Chapter 2 - Page 8

f. Return to your monitoring script output. Confirm that the number of active sessions hasincreased by 10.

Also, depending on the responsiveness of the server and database, you may witnessthe number of data source connections increase as well.

g. Kill the monitoring script.


15/128



Chapter 2 - Page 9


1. Launch the Lab Framework command shell by executing the


2. Change the current directory to .

3. Execute the following:

ant setup_solution

The Lab Framework performs the same tasks as described in the “Initialize theMedRec server infrastructure” section.

4. The solution monitoring script can be found at

/solution/monitorMedRec.py.


16/128



Chapter 2 - Page 10


17/128



Chapter 3 - Page 1


Chapter 3


18/128



Chapter 3 - Page 2


Practices Overview

In this practice you will use Guardian to evaluate an existing domain.

Naming Conventions

In addition to the variable names listed earlier, these practices use the following variable nameto refer to a frequently used location on your file system. This value is also defined as a Linuxenvironment variable.

Variable Path

/u01/app/oracle/Guardian/10.3.2


19/128



Chapter 3 - Page 3

Practice 3-1: Using Guardian to Evaluate a Domain


Skills Learned


• Update Guardian with newer problem signatures

• Deploy the Guardian agent to a domain• Activate and inventory a domain

• Use Guardian to evaluate a domain for problems

• Correct a problem identified by Guardian

Overview

Regularly running Oracle Guardian allows you to quickly detect and fix common problems withyour domain’s configuration. Guardian consists of two parts: a graphical interface and an agentthat runs on every server in your domain.

In this practice, you activate the Guardian agent within your domain, and then use the Guardiantool to connect to and evaluate your domain. You also use this opportunity to correct a problemdetected by Guardian. Normally, you would use your Oracle Support account to download thelatest Guardian problem signatures from the Web, but due to the constraints of the labenvironment, a signature pack has already been downloaded for you.

Tip: Unless otherwise indicated, all labs require the MedRecDomain domain. If you do not havethis domain, follow the Solution Instructions for the “Developing a Custom Monitoring Script”practice.

Instructions

1. Initialize the lab environment.

a. Start your administration server, if not already started.

b. Launch a Lab Framework command shell, if not already open.

c. Change the current directory to .

d. Execute the following:ant setup_exercise

The Lab Framework updates the MedRecDomain domain with a configuration problemthat Guardian will later detect.

2. Update and launch Guardian.

a. Locate the /repository/archives directory. Rename the fileweblogic-signatures.jar to weblogic-signatures.jar.old.

b. Locate the /resources/weblogic-signatures.jar file. Copythis file.

c. Paste the newer signature file into the

/repository/archives directory.Why? You are replacing the signature file that came with the Guardian installation witha later signature file.

d. Launch a Terminal and navigate to .

e. Execute the following:

export PATH=$MIDDLEWARE_HOME/jdk160_18/bin:$PATH

./guardian


20/128



Chapter 3 - Page 4

f. When prompted, enter the following new Workspace:

/guardianWorkspace. Then click Finish.

Tip: Replace with the actual path.

3. Deploy the Guardian agent.

a. Launch a Web browser and access the WLS administration console:

http://localhost:7020/console

b. Log in as weblogic/Welcome1 and Lock the console.

c. In the Domain Structure panel, click the domain name, MedRecDomain.

d. Select the Enable Oracle Guardian Agent check box and click Save.

e. Activate your changes.

f. Kill (Ctrl + c) your administration and managed servers.

g. Launch two Terminals and restart the servers:

cd /home/oracle/wls11g_trouble/work/domains/MedRecDomain

./startWebLogic.sh

cd /home/oracle/wls11g_trouble/work/domains/MedRecDomain

./startServer1.sh

Tip: Alternatively, open two tabs within a single Terminal window.Tip: To avoid supplying credentials each time a server is started, recall that you canoptionally create a boot.properties file for each of your servers.

4. Activate and inventory a domain.

a. Return to the Guardian application.

b. On the main toolbar, click Activate.

c. Enter the location and credentials of your administration server and click Finish.

d. An initial domain inventory is generated and displayed. Browse the contents of the

Servers and JDBC sections.5. Evaluate a domain.

a. Click the Evaluate button on the toolbar.

The Evaluation Wizard appears.

b. Confirm that MedRecDomain is selected and enter the Domain Credentials.

c. In the Bundle column, select the All Signatures option.


21/128



Chapter 3 - Page 5

d. Select the Create Shortcut check box. Click Finish.

When the evaluation finishes, an evaluation summary is generated and displayed.

e. Confirm that you have at least one Detected Signature. For example:

f. Locate and select the signature concerning HTTP POST.

Tip: To avoid scrolling, maximize the evaluation summary panel.

g. Read the details of this problem signature in the Description section.

If time permits at the end of the practice, you can read about any remaining detectedsignatures as well.

6. Resolve a detected problem.

a. Return to the console and Lock it.

Tip: You will have to log in again since the admin server has been restarted.

b. Locate and select the MedRecSvr1 server.

c. Click the Protocols > HTTP tab.

d. Set the value of Max Post Size to 1048576. Click Save.

Why? The value of -1 means an unlimited post size. You are setting it to a morereasonable 1 MB.

e. Repeat the previous steps to update the Max Post Size of the MedRecAdmSvr server.

f. Activate your changes.

Note: You may ignore any errors in the server windows.

g. Kill and restart your servers.

h. Return to Guardian.

i. Select File > Close All.

j. Click the Shortcut Explorer tab.

k. Double-click the shortcut for MedRecDomain.

l. Perform another evaluation. Verify that the HTTP POST (MaxPostSize) problem is nolonger detected.

m. When you are finished with the practice, do the following:


22/128



Chapter 3 - Page 6

1) Exit Guardian.

2) Use the console to disable the Guardian agent for the domain.

3) Restart both servers.


23/128



Chapter 3 - Page 7


There is no solution for this practice. If you only partially completed the practice instructions,however, then you need to undo what you have done by following these steps:





ant setup_solution

The Lab Framework:

− Makes a backup copy of your current work

− Starts database and WebLogic servers, if not already started

− Disables the Guardian agent

− Updates server protocol settings

− (Re)deploys the MedRec application


24/128



Chapter 3 - Page 8


25/128



Chapter 4 - Page 1


Chapter 4


26/128



Chapter 4 - Page 2


Practices Overview

In these practices you will set up WebLogic Diagnostic Framework harvesters, watches, andnotifications. You will also use the Monitoring Dashboard to view diagnostic information ascharts and graphs.


27/128



Chapter 4 - Page 3

Practice 4-1: Harvesting Diagnostic Metrics


Skills Learned


• Setup an existing JMS queue for diagnostic notifications

•

Create a system diagnostic module• Configure metric collectors for several MBean types

• Trigger notifications under various MBean watch conditions

• Access diagnostic archives and inspect recorded metrics

Overview

Patients have reported intermittent problems with the MedRec application. Before contacting thedevelopment team and starting to troubleshoot, however, the IT group would like to get moreinsight into the server and application. The WebLogic Diagnostics Framework (WLDF) allowsyou to collect and record server and application metrics over time for later analysis. WLDF alsosupports watch conditions that can trigger notifications, such as posting a JMS message orsending an SNMP trap.

In this practice, you configure a new system-wide WLDF module along with several metriccollectors (harvesters), watches, and notifications. The diagnostic configuration is shown in thefollowing diagram:

Instructions

1. Create a JMS queue for diagnostic notifications.

a. Copy the following file to /client:

/resources/runclients.sh

Overwrite any previous version of the file.

b. Launch a Lab Framework command shell, if not already open.

c. Navigate to /resources and execute the createWLDFQueue.py WLST script.

d. Access the admin console.

e. From the Domain Structure panel, select Services > Messaging > JMS Modules.

f. Select the MedRec-jms module.


28/128



Chapter 4 - Page 4

g. Confirm that a new queue is available named WLDFNotificationQueue. Note its JNDIName:

2. Create a system diagnostic module.

a. Lock the console.

b. From the Domain Structure panel, select Diagnostics > Diagnostic Modules.

c. Click New.

d. Name the module MedRecDiagnostics and click OK.

e. Edit the new module.

f. Click the Targets tab.g. Select MedRecSvr1 and click Save.

3. Define a metric collector (harvester).

a. Click the Configuration > Collected Metrics tab.

b. Set the Sampling Period to 30000 (30 seconds) and click Save.

c. Click the New button.

d. Click Next.

Note: Since this diagnostic module is targeted to a managed server, only the ServerRuntime MBeans apply.

e. From the select box, select the weblogic.management.runtime.

JDBCConnectionPoolRuntimeMBean option. Click Next.f. Move the following attributes from the Available column to the Chosen column:

ActiveConnectionsCurrentCount

CurrCapacity

LeakedConnectionCount

Click Next.

g. Note the available values for Collected Instances, but make no selection. ClickFinish.

Note: If you do not identify specific MBean instances, WLDF collects the selectedattributes from all available instances.

h. Click New once again.i. Repeat this process to define the following additional metric:

Field Value

MBean Server Location ServerRuntime

MBean Type weblogic.management.runtime.WebAppComponentRuntimeMBean


29/128



Chapter 4 - Page 5

Collected Attributes OpenSessionsCurrentCount

Collected Instances Select the option that contains the text:Name=MedRecSvr1/medrec,ServerRuntime=MedRecSvr1

4. Create a diagnostic JMS notification.

a. Click the Configuration > Watches and Notifications tab.

b. Click the Notifications tab at the bottom of the screen.

c. Click New.

d. For the Type field, select the JMS Message option. Click Next.

e. Enter JMSNotification as the Notification Name, and click Next.

f. Enter the following values:

Field Value

JMS Destination JNDI Name com.medrec.jms.WLDFNotificationQueue

Connection Factory JNDIName

weblogic.jms.XAConnectionFactory

Click Finish.

Tip: Be sure you type these values exactly as shown.

5. Create diagnostic watches.a. Click the Watches tab at the bottom of the screen. Then click New.

b. Enter JDBCWatch for the Watch Name, and confirm that the Watch Type is CollectedMetrics. Click Next.

c. Click the Add Expressions button.

d. Click Next.

e. Select the same JDBCConnectionPoolRuntimeMBean that is being collected andclick Next.

f. Click Next to watch all instances of this MBean.

g. Enter the following values:

Field Value

Message Attribute ActiveConnectionsCurrentCount

Operator >

Value 1

Click Finish.


30/128



Chapter 4 - Page 6

h. Click Next.

i. Select the Use a Manual Reset Alarm option and click Next.

j. Move the JMSNotification option from the Available column to the Chosen column.Click Finish.

k. Repeat this process to configure a second watch for the Web application metric beingcollected:

Field Value

Watch Name SessionWatch

Watch Type Collected Metrics

MBean Server Location ServerRuntime

MBean Type weblogic.management.runtime.WebAppComponentRuntimeMBean

Instance Select the option that contains the text: Name=MedRecSvr1/medrec,ServerRuntime=MedRecSvr1

Message Attribute OpenSessionsCurrentCountOperator >

Value 10

Config Watch Alarm Use a manual reset alarm

Chosen JMSNotification

l. Activate your changes.

6. Trigger WLDF notifications.

a. From a Lab Framework prompt, execute /client/runclients.sh.

Note: The test clients require a few minutes to finish. Recall that the metric collector is

running every 30 seconds.

b. From the administration console, return to the MedRec-jms JMS module.

Tip: Remember that you can return to a prior screen by using the breadcrumb trail atthe top of the console.

c. Click the WLDFNotificationQueue.

d. Click the Monitoring tab.

e. Confirm that at least one message has been added to the queue.

Tip: If there are no current messages try running the test client again.

f. Locate the shell running MedRecSvr1. Confirm that the watch notification was alsowritten to the server log. For example:


31/128



Chapter 4 - Page 7

... Collected Metrics tab, uncheck Enabled.

j. Similarly, disable the watch and notification component of the module.

k. Activate your changes.

7. Browse collected metrics by using the admin console.

a. From the Domain Structure panel, select Diagnostics > Log Files.

b. Locate and select the HarvestedDataArchive log file for the MedRecSvr1 server.Click View.

c. Click the Customize this table link.

d. For Time Interval, select Last 15 Minute(s).

e. Under Column Display, update the Chosen column to contain only these fields:

Date

Attribute

Value

f. For Number of rows displayed per page, select 100. Click Apply.

g. Browse the harvester entries. Verify that the four MBean attributes are being collectedapproximately every 30 seconds.

h. Click Customize this table again.

i. For WLDF Query Expression, enter the following:

ATTRNAME = 'CurrCapacity'

Click Apply.

Tip: Be sure to use single quotes.

j. Confirm that now only this attribute is displayed.

k. Update the WLDF Query Expression again and replace it with the following:

TYPE LIKE '%WebAppComponentRuntimeMBean'

Tip: Be sure to use single quotes.

l. Confirm that now only the OpenSessionsCurrentCount attribute is displayed.


32/128



Chapter 4 - Page 8


1. If the /domains/MedRecDomain location does not yet exist, follow theSolution Instructions for the “Developing a Custom Monitoring Script” practice.

2. Launch the Lab Framework command shell by executing the file

/bin/prompt.sh.



ant setup_solution

The Lab Framework:


− Creates a JMS queue to receive WLDF notifications

− Creates a diagnostic module

− Creates metric collectors for two MBean types

− Creates a diagnostic notification

− Creates a diagnostic watch

Note that all solution WLDF components will be disabled by default.


33/128



Chapter 4 - Page 9

Practice 4-2: Monitoring Diagnostic Metrics


Skills Learned


• Access the Monitoring Dashboard

•

Monitor standard views in the Monitoring Dashboard• Define a custom view in the Monitoring Dashboard

• Configure charts within a view

Overview

The WebLogic Server administration console provides a link to a dashboard that allows you tomonitor diagnostic metrics and events. Values can be viewed over time as charts. Collections ofrelated charts are known as views. In this practice, you experiment with the MonitoringDashboard and monitor MBean attributes. The Monitoring Dashboard architecture is shown inthe following diagram:

Instructions

1. Monitor standard views.

a. Launch an admin console and log in.

b. On the Home page, click the Monitoring Dashboard link (under Charts and Graphs).

Tip: You may also enter the URL for the dashboard directly into a Web browser:http://localhost:7020/console/dashboard

c. In the left panel, under the View List tab, locate the JVM Runtime Heap under Built-in Views. Select it.

d. Then click the green Start button above the tabs to begin collecting data.e. Locate a Lab Framework command prompt or start a new one.

f. Run /client/runclients.sh to test the MedRec application.

g. Return to the Monitoring Dashboard and view the heap statistics for MedRecSvr1. Forexample:


34/128



Chapter 4 - Page 10

h. Click the Stop All button to stop collecting this view’s MBean metrics.

2. Reactivate the metric collector.

a. Return to the administration console and Lock it.

b. Locate and edit your existing diagnostics module, MedRecDiagnostics.c. Click the Configuration > Collected Metrics tab.

d. Select the Enabled check box and click Save.

e. Activate your changes.

f. Kill and restart MedRecSvr1 to reset its runtime MBean attributes.

g. To create more interesting data, from a Lab Framework prompt execute

/client/runclients.sh.

3. Define a custom view for harvested metrics.

a. Return to the Monitoring Dashboard.

b. In the left panel, click the My Views folder. Click the drop-down menu arrow (below

the tabs) and select New View. Type over the default name of the new view with:MedRec Diagnostics.

c. Ensure the new view is selected and click the Metric Browser tab.

d. In the Servers drop-down list, select MedRecSvr1, select Collected Metrics Only,and click the Go button.

e. In the Types list of available MBeans on this server, locate and selectJDBCConnectionPool.

f. Under Instances, select MedRecGlobalDataSourceXA.

g. Under Metrics, drag and drop the CurrCapacity (int) attribute onto the right panel. Anew chart is created. To change the chart’s default name, click the menu down-arrow

in the chart title bar and select Properties. For Chart Title enter JDBC Chart and

click OK.Tip: If you see no data points in the chart, then you must set the time of the chart toinclude when the metrics were actually collected (rather than the default time):

1) Go to the admin console.

2) Under Diagnostics > Log Files, select HarvestedDataArchive forMedRecSvr1 and click View. Click Customize this table. For WLDF Query

Expression, enter: ATTRNAME = 'CurrCapacity' and click Apply.


35/128



Chapter 4 - Page 11

3) Make note of the information in the Date field of the first CurrCapacity Attribute.

4) Return to the Monitoring Dashboard.

5) At the top-right of the chart, click the down arrow for the menu and selectProperties.

6) Under Time Range select Custom and enter the Start Time. Use the correct

format. For example: in the log file if the Date field was "02/11/11 18:50:41820" you would enter it here as: "02/11/2011 6:50:41 PM"

Tip: Do not include the double quotes.

7) Enter a Duration of 5 minutes: 0 00:05

8) Click OK.

9) You may wish to zoom out by clicking the Zoom Out button.

h. Also drag and drop the ActiveConnectionsCurrentCount (int) attribute onto thesame chart to create a second graph in a different color.

4. Work with charts and graphs in real time.

a. In the Monitoring Dashboard, click the View List tab and create another custom view.Name it MedRec Sessions. Select the new, empty view.

b. Click the Metric Browser tab. In the Servers drop-down list, select MedRecSvr1,make sure Collected Metrics Only is NOT checked, and click the Go button.

c. In the Types list of available MBeans on this server, locate and selectWebAppComponent.

d. Under Instances, select MedRecSvr1_/medrec, medrec.

e. Under Metrics, drag and drop the OpenSessionsCurrentCount (int) attribute ontothe right panel.

f. Click the menu down-arrow in the chart title bar and select Properties. For Chart Title enter Sessions Chart and click OK.

g. Retest the MedRec application by using runclients.sh. While the script is stillrunning return to the Monitoring Dashboard.

h. Click the green Start button and collect data for a few minutes. Re-run the clientsscript if you wish.

Tip: Mouse over a specific data point to view its value and timestamp.

5. Deactivate metric collection.

a. Stop All views, if any are still running.

b. Close the Monitoring Dashboard.

c. Return to the administration console and disable your diagnostic module’s metriccollector once again.


No solution exists for this practice.


36/128



Chapter 4 - Page 12


37/128



Chapter 5 - Page 1


Chapter 5


38/128



Chapter 5 - Page 2


Practices Overview

In these practices you will set up application-level instrumentation and dye masks to filterdiagnostic events.


39/128



Chapter 5 - Page 3

Practice 5-1: Configuring and Monitoring Diagnostic Events


Skills Learned


• Use the console to generate and edit a deployment plan

•

Use the console to configure application instrumentation• Configure built-in diagnostic monitors

• Define pointcuts for custom monitors

• Inspect events found in diagnostic archives

Overview

A few patients have reported performance problems logging into the MedRec application.Before contacting the development team, however, you would like to gain more knowledgeabout how the MedRec login process works behind the scenes and also how it performs. TheWLDF Instrumentation component allows you record diagnostic data when certain events occurwithin WLS or your application code. For example, you can automatically record the methodarguments or stack trace whenever a specific Java class is executed.

While some built-in instrumentation monitors can be configured at the server level, most built-inmonitors and all custom monitors must be defined by using application deployment descriptors.

In this practice, you create a basic diagnostic module and include it within the MedRecapplication. You then utilize a deployment plan to dynamically add diagnostic monitors to thismodule and to enable/disable them as needed. This practice’s diagnostic configuration is shownin the following diagram:

Instructions

1. Enable instrumentation on the server.

a. If your domain does not include a diagnostic module, complete the SolutionInstructions for the “Harvesting Diagnostic Metrics” practice.

b. Launch the console and Lock it.

c. Navigate to your diagnostics module, MedRecDiagnostics.

d. Click the Configuration > Instrumentation tab.

e. Select the Enabled check box.

f. Save and Activate your changes.

2. Add a WLDF descriptor file to an application.

a. Inspect the contents of the file /resources/META-

INF/weblogic-diagnostics.xml.


40/128



Chapter 5 - Page 4

b. Launch a Lab Framework prompt and navigate to /resources.

c. Execute the following:

jar uf /applications/medrec.ear META-INF

d. Using jar or the Linux Archive Manager, inspect medrec.ear and confirm that thefile was added.

e. Return to the console and Lock it.

f. From the Domain Structure panel, click Deployments.

g. Update the medrec application and Activate your changes.

3. Generate a WLDF deployment plan by using the console.

a. From the same Deployments table, select medrec.

b. Click the Configuration > Instrumentation tab.

c. Lock the console and click the Add Monitor From Library button.

d. Move the EJB_Before_SessionEjbBusinessMethods option from the Available column to the Chosen column. Click OK.

Because a deployment plan does not yet exist for this application, the SaveDeployment Plan Assistant is launched.

e. Set the Path to /applications/medrec-plan.xml. Click OK.

4. Configure a standard EJB monitor.

a. Return to the Configuration > Instrumentation tab and select your new monitor.

b. Under Actions, move the StackDumpAction option from the Available column to theChosen column.

c. Click Save. Activate your changes.

d. Return to the medrec application’s main configuration page and inspect the Overview tab. Confirm that a Deployment Plan is now assigned to the application.

e. Inspect the generated plan file. The plan should dynamically add a new wldf-instrumentation-monitor element to the weblogic-diagnostics.xml

descriptor, whose child elements resemble the following:

Element Value

name EJB_Before_SessionEjbBusinessMethods

action StackDumpAction

Tip: The console also generates a folder /applications/plan, whichyou can delete.

5. Test the application and inspect the generated EJB events.

a. From a Lab Framework prompt, execute /client/runclient.sh.

Why? This script logs a user into the patient application and then views his/her history.

b. Return to the admin console. From the Domain Structure panel, select Diagnostics >Log Files.

c. Locate and select the EventsDataArchive log file for the MedRecSvr1 server. ClickView.

d. Click the Customize this table link. Under Column Display, remove from Chosen thecolumns Context ID, User ID, and Payload.

Tip: The Payload column includes the stack trace at the time of the event. Laterpractices focus on analyzing stack traces.


41/128



Chapter 5 - Page 5

e. Click Apply.

f. Browse the generated events.

g. Locate events for the

com.bea.medrec.repository.impl.PatientServiceImpl class. Whichmethod do you think corresponds to a user logging in?

Now we create a custom monitor that just focuses on this code.

6. Define a custom monitor.a. Lock the admin console and once again edit the medrec application. Go to the

Configuration > Instrumentation tab.

b. Edit the EJB_Before_SessionEjbBusinessMethods monitor. Clear the Enabled check box and click Save.

c. Use the browser’s back button or the console’s breadcrumb links to return back to themain Configuration > Instrumentation tab.

d. Click the Add Custom Monitor button.

e. Enter the following values:

Field Value

MonitorName

DebugPatientService

LocationType

Around

Pointcut execution(* *impl.PatientServiceImplauth*(...))

Click OK.

Tip: This pointcut expression includes all calls to methods that start with the nameauth in classes that end with the name impl.PatientServiceImpl.

f. Select your new monitor.

g. Under Actions, move the TraceElapsedTimeAction option from the Available column to the Chosen column. Click Save.

h. Update the medrec application again and Activate your changes.

i. Inspect the plan file again and confirm that it now defines two monitors.

7. Analyze the application.

a. Execute runclient.sh again.

b. Repeat the previous steps to view the latest data in EventsDataArchive.

c. Add the Payload column back to the list of displayed attributes.

d. Locate the events from the custom DebugPatientService monitor.

Each TraceElapsedTime action generates two events, one before the method

execution and one after. The latter event includes the total time (in nanoseconds) toperform the authentication.


42/128



Chapter 5 - Page 6



2. Launch the Lab Framework command shell /bin/prompt.sh file.


4. Execute the following:ant setup_solution

The Lab Framework:


− Runs the solution for the “Harvesting Diagnostic Metrics” practice.

− Enables instrumentation on the WLDF module

− Deploys an updated MedRec application along with a deployment plan

Note that all application-scoped WLDF components will be disabled by default.


43/128



Chapter 5 - Page 7

Practice 5-2: Tracing a Client Request


Skills Learned


• Enable server debug messages

•

Use context IDs to associate log entries with a request• Configure a Dye Injection monitor

• Filter recorded events by using dye masks

Overview

When instrumentation is enabled on a diagnostic module, all WLDF events and log entriesassociated with a client request will be tagged with a unique context ID. These IDs allowadministrators to trace all the diagnostic data for a specific request.

In some troubleshooting cases, you may wish to go a step further and limit the conditions underwhich WLDF events are recorded. A diagnostic context ID supports various flags or “dyes” thatallow you to filter the types of requests that trigger instrumentation data.

In this practice, you use the console to locate context IDs and try to map them to specific

MedRec application functionality. You also use this opportunity to experiment with the dyeinjection monitor and dye masks. The lab infrastructure is shown in the following diagram:

Instructions

1. Generate debug messages in the server log.

a. If your domain does not include a diagnostic module, complete the SolutionInstructions for the “Configuring and Monitoring Diagnostic Events” practice.

b. Launch the console and Lock it.

c. Locate and edit MedRecSvr1.d. Click the Debug tab.

e. Locate the weblogic > servlet > internal debug scope.


44/128



Chapter 5 - Page 8

f. Select the check box for the internal scope and click the Enable button.

g. Similarly, enable the weblogic > ejb > invoke debug scope.

h. Activate your changes.

2. View context IDs for WLDF records.

a. Launch a second browser window and access the MedRec application:

http://localhost:7021/medrec/index.action

b. Under the Patient section, click the Login link.c. Log in as the user [email protected]/weblogic. Leave this browser session open.

d. Return to the console and view EventsDataArchive for MedRecSvr1.

e. Click Customize this table.

f. Add the Context ID column to the chosen list, if not already chosen. Click Apply.

g. Locate the Context ID for the latest event generated by your customDebugPatientService monitor. For example:

Tip: To avoid scrolling, under Customize this table you may want to reorder thecolumns so that Context ID and Class are displayed adjacently.

h. Copy the last set of non-zero digits. In the example above, that would be “1e58c”.

3. Correlate WLDF and server log messages.

a. Use the console to view the ServerLog for MedRecSvr1.

b. Once again, add the Context ID column to the chosen list of columns. Also increasethe Number of Rows Displayed Per Page to 1000.

c. Browse the recent log entries. Notice that some entries have been assigned context

IDs while others have not. The latter are internal messages not associated with a clientrequest.

d. Click Customize this table again.

e. For WLDF Query Expression, enter the following:

CONTEXTID LIKE '%nnnnn'

For nnnnn, paste the context ID fragment that you copied from the WLDF event.

Tip: If you do not see any entries, click Customize this table again and change TimeInterval to Last 15 minute(s) (or longer).


45/128



Chapter 5 - Page 9

f. Click Apply.

g. Browse all the server debug messages associated with this login request. TheSubsystem column should indicate “Http,” “HttpSessions,” and “EjbInvoke.”

h. Using these log messages, try to answer the following question: Upon logging into theMedRec application, what is the name of the JSP file that executes?

i. Return to the MedRec application. Click the Successfully logged in! Click here tocontinue link. Then click one of the listed patient Records.

j. Note the URL in the browser and the current time.

k. Access the server log again and view the latest entries. You must remove your customlog filter criteria.

l. Search the Web page contents for the text “viewRecordSummary.” Try to determinethe context ID for the latest MedRec client request.

m. Once again, use the WLDF Query Expression field to show only those log messagesfor this context ID. Browse the contents of these messages.

n. Edit your MedRecSvr1 configuration again and Disable all debug attributes.

Tip: You can use the check box in the table header to select all other check boxes atonce.

4. Configure the dye injection monitor to flag an IP address.

a. Obtain the IP address of your Linux VM instance. From a Linux terminal, execute:

/sbin/ifconfig.

Tip: If your VM is bound to multiple IP addresses, you can use the previouslygenerated debug messages to determine the actual IP being used. It is probably thefirst one listed. If you use the wrong one you will have to repeat these instructions andtry the other ones.


46/128



Chapter 5 - Page 10

b. Return to the console and Lock it.

c. Locate and edit your system diagnostic module, MedRecDiagnostics.

d. Click the Configuration > Instrumentation tab.

e. Click the Add/Remove button.

f. Move the DyeInjection monitor from the Available column to the Chosen column.Click OK.

g. Edit the new monitor.

h. Under Properties, enter the following:

ADDR1=your_IP_address

i. Click Save. Activate your changes.

5. Configure dye filtering for a monitor.

a. Lock the console again.

b. Edit the medrec application.

c. Click the Configuration > Instrumentation tab.

d. Click Add Monitor from Library.

e. Add the monitor Servlet_Before_Session and click OK.

f. Edit the new monitor. Enter or select the following values:

Field Value

Enabled

Actions (Chosen) TraceAction

EnableDyeFiltering

Dye Mask (Chosen) ADDR1

Click Save.

g. Update the medrec application. Activate your changes.

6. Test event filtering.a. Retest the MedRec application as done previously.

b. Return to the console and view EventsDataArchive for MedRecSvr1.

c. Using the Monitor column, confirm that events were recorded from theServlet_Before_Session monitor.

Tip: If the events are not shown, try setting the dye flag to another available IPaddress.

d. Edit the MedRecDiagnostics module and select the DyeInjection monitor.

e. Edit this monitor’s Properties. Change the ADDR1 flag to use some other arbitrary IPaddress.

Tip: Alternatively, use another student’s IP address.

f. Update the medrec application. Activate your changes.

g. Test the application a final time.

h. In the admin console, view the latest event data. Using the records’ timestamps, verifythat there are no new events from the Servlet_Before_Session monitor.

i. When you are finished with the practice, edit MedRecDiagnostics and disable theinstrumentation component.


47/128



Chapter 5 - Page 11







ant setup_solution

The Lab Framework:


− Runs the solution for the “Configuring and Monitoring Diagnostic Events” practice

− Disables debug flags, if enabled

− Adds a new dye injection monitor to the WLDF module

− Deploys the MedRec application by using an updated deployment plan

Note that all WLDF components will be disabled by default.

Note that the dye injection monitor will be configured with a dummy IP address.


48/128



Chapter 5 - Page 12


49/128



Chapter 6 - Page 1


Chapter 6


50/128



Chapter 6 - Page 2


Practices Overview

In these practices you will troubleshoot the Sun JVM by using a heap profile. You will alsotroubleshoot the JRockit JVM by using JRockit Mission Control.


51/128



Chapter 6 - Page 3

Practice 6-1: Troubleshooting a Running JVM


Skills Learned


• Start a server running in the Sun JVM

•

Perform and analyze a server thread dump• Configure JVM heap settings

• Generate a heap profile by using the Sun JVM

• Analyze a heap profile

Overview

Two of the most common troubleshooting scenarios in server software are server hangs (or veryhigh response times) and memory leaks. Servers can hang for several reasons, includingdeadlocks and full CPU utilization. Memory leaks are less common in Java than otherlanguages, but can occur when applications deliberately or accidentally create too muchgarbage (objects), or make objects ineligible for garbage collection by the JVM. For example,developers may place too many objects in the HTTP session scope, which by default are not

eligible for garbage collection for at least 1 hour.WebLogic Server and the underlying JVM provide several tools to help troubleshoot theseproblems. In this practice, you investigate an application that seems to be hung, and use JVMthread dumps to help determine potential causes. Similarly, you investigate a possible memoryleak with the aid of a JVM-generated heap profile. The lab infrastructure is depicted in thefollowing diagram:

Instructions

1. Set up the practice.

a. Locate a Lab Framework prompt or start a new one. Change directories to

.

b. Execute the following:

ant setup_exercise

The Lab Framework deploys a new version of the MedRec application.c. Kill MedRecSvr1.

d. Restart the server by using the following commands:

export JAVA_VENDOR=Sun

./startServer1.sh

e. Use the initial server output to confirm that the Sun JVM is being used:

Java(TM) SE Runtime Environment (build ...)


52/128



Chapter 6 - Page 4

Java HotSpot(TM) Server VM (build ...)

f. From another Linux terminal enter the following:

ps –ef | grep MedRecSvr1

g. Make a note of the process ID for MedRecSvr1. For example:

oracle 31578 31539 ... java -server -Xms256m -Xmx512m -XX:MaxPermSize=128m -Dweblogic.Name=MedRecSvr1 ...

2. Perform a thread dump under load.

a. From the Lab Framework prompt, navigate to /client.

b. Execute the runclients.sh script.

You will notice that the test client and, therefore, the application as well, seem to betaking an unusually long time to respond.

c. While the test client continues to run, signal the MedRecSvr1 process to perform athread dump:

kill -3

d. Locate the output for MedRecSvr1. Confirm the presence of a thread dump. For

example:Full thread dump Java HotSpot(TM) Server VM

...

Heap

...

e. Locate one or more threads that are currently running the MedRec application. Forexample:

"[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default(self-tuning)'" ...

...

at com.bea.medrec...f. From these thread details, can you determine which MedRec Java class or method

may be responsible for the slow application?

Tip: For those with some Java knowledge, the culprit code can also be found at/resources.

g. Allow the test client to finish, if still running.

Tip: If you get tired of waiting, press Ctrl + C in the window where it is running.

3. Trigger an "Out of Memory" error.

a. Execute the /client/runadminclients.sh script.

This client tests a different portion of the MedRec application. Specifically, the part ofthe application that allows administrative users to manage patient accounts.

b. Locate the output for MedRecSvr1 and verify that an OutOfMemoryError has beengenerated similar to the following:

... java.lang.OutOfMemoryError: Java heap space ...

c. Use the stack trace to identify some possible culprits for the high memory usage in the

MedRec application (com.bea.medrec.*). For example:

Caused By: ...


53/128



Chapter 6 - Page 5

at com.bea.medrec.web.controller.

ViewingNewlyRegisteredPatientsController

...

4. Increase the server heap.

a. Kill MedRecSvr1.

b. Restart the server by using the following additional environment variable:

export USER_MEM_ARGS="-Xms512m –Xmx768m –XX:MaxPermSize=128m"

Tip: If you use a new terminal, you must set the JAVA_VENDOR variable again.

c. Inspect the initial server output to confirm the new server heap settings. For example:

JAVA Memory arguments: -Xms512m –Xms768m ...

Tip: You can also view server JVM settings by using the administration console.

d. Execute runadminclients.sh again.

e. Verify that this time no OutOfMemoryError messages were generated.

5. Enable the Sun JVM memory profiler.

a. Kill MedRecSvr1 again.

b. Restart the server by using the following additional environment variable:

export JAVA_OPTIONS=-agentlib:hprof=heap=sites,interval=100

Important: Do not miss the dash (“-”) before agentlib.

Tip: The server requires a few minutes longer to boot as the agent initializes andbegins collecting data.

c. After the server is up, once again determine its current process ID, for later use.

6. Generate a heap profile.

a. Execute runadminclients.sh again.

b. After the test finishes, issue the kill -3 command to the MedRecSvr1 process.

c. In addition to a thread dump, the JVM begins to create a heap profile:...

Dumping allocation sites ...

d. The profile requires some time to capture, perhaps as long as 5 minutes or more.When it is finished, the JVM outputs “Done.”

7. Analyze profiler output.

a. View the MedRecDomain/java.hprof.txt file.

b. At the end of the file, locate the SITES BEGIN section. Find the top ranked classnames in terms of heap consumption. For example:

SITES BEGIN...

percent ... stack class rank self accum trace name

1 92.32 92.32% 408543 java.lang.Byte[]

2 ...

c. Locate the stack trace number for this class name. Search the file for this number tolocate the complete trace entry. For example:

TRACE 408543:


54/128



Chapter 6 - Page 6

...

d. Use the stack trace entry to determine which part of the application is responsible forcreating these large objects.

Tip: Once again, the actual culprit code can also be found at/resources, for reference.

Bonus Instructions

Launch Sun’s JConsole or JVisualVM tools (found at /jdk.../bin).Restart the server and run the test again while monitoring various JVM metrics, such as heapsize and garbage collection times. To connect to a JVM, you must know its process ID.


No solution exists for this practice. To do this practice the MedRecDomain must exist. If the

/domains/MedRecDomain location does not yet exist, follow the SolutionInstructions for the “Developing a Custom Monitoring Script” practice.


55/128



Chapter 6 - Page 7

Practice 6-2: Troubleshooting Applications on JRockit


Skills Learned


• Start a server running in the JRockit JVM

•

Navigate the features of JRockit Mission Control• Monitor CPU utilization by using the Management Console

• Profile execution time by using the JRockit Flight Recorder

• Profile heap usage with the Memory Leak Detector

Overview

In addition to providing a durable, high performance JVM, Oracle JRockit includes manytroubleshooting and diagnostic tools, including JRockit Mission Control. Using Mission Control,you can monitor the health and performance of a JVM in real time by using customizable graphsand gauges. You can also record performance and memory profiles for later analysis. Finally,Mission Control includes software to help identify and troubleshoot possible memory leaks onthe server. In this practice, you utilize all of these capabilities to investigate several application

problems.The general Mission Control architecture for this practice is shown in the following diagram:

Instructions1. Set up the practice.


.


ant setup_exercise

The Lab Framework deploys a new version of the MedRec application.

2. Start JRockit Mission Control.

a. Kill MedRecSvr1 by using the kill -9 command (to avoid the generation of anyadditional heap profiles).

b. Restart the server by using the following commands:

export USER_MEM_ARGS="-Xms512m –Xmx768m"

export JAVA_VENDOR=Oracle

export JAVA_OPTIONS=-Xmanagement

./startServer1.sh

c. In the server’s initial output, confirm that the following message is written:

[INFO][mgmnt ] Local JMX connector started


56/128



Chapter 6 - Page 8

d. From a Linux terminal enter the following:

ps –ef | grep MedRecSvr1

Make a note of the process ID for MedRecSvr1.

e. Execute the file /jrockit.../bin/jrmc.

Tip: The exact name of the JRockit root folder depends on the version and releasenumber that is installed in your environment.

f. In the JVM Browser panel or “view,” locate the Java process for MedRecSvr1. Ifnecessary, cross reference the process ID located in parentheses.

3. Monitor CPU utilization by using the Management Console.

a. Right-click the server in the JVM Browser view and select Start Console.

A console view appears to the right. Resize the views as necessary so that the CPUgauge is displayed:

b. From the Lab Framework prompt, navigate to /client.

c. Execute the runbasicclients.sh script.

d. While the test client is running, return to Mission Control and monitor the CPU andHeap utilization.

This simple login test should not yield any extraordinary results, and should finishwithin a minute.

e. Now execute the runclients.sh script.

f. Using Mission Control, confirm that during this new test the CPU gauge approaches orbecomes 100%.

g. You can also view the historical CPU utilization by using the Processor section found

below the gauges:


57/128



Chapter 6 - Page 9

h. The test requires a few minutes to finish. When finished, close the console view forMedRecSvr1 in JRockit Mission Control (JRMC), but do not close JRMC.

4. Profile execution time by using the JRockit Flight Recorder (JFR).

a. Right-click the server in the JVM Browser view and select Start Flight Recording.

b. Notice the location of the flight recording file shown in the Filename field. Ensure Timefixed recording is selected. For Recording Time, enter 3 min. Click OK.

c. While the Flight Recorder executes, start the runclients.sh script a second time.

d. When the Flight Recorder finishes after 3 minutes, use the File menu to open the file.

File > Open File. Select the .jfr file (under /home/oracle) and click OK.

e. General > Overview is displayed by default. Inspect the Live Set, HeapFragmentation, and GC Pause Time gauges.

f. Click the Code button on the left side of the view.

g. Inspect the Hot Packages and Hot Classes sections under the Overview tab. Thedata should indicate that the MedRec application is likely the reason for the high CPUresults. For example:

h. Click the Hot Methods tab at the bottom of the view.

i. The supplied table lists the code that was executed the most frequently. For example:

j. Select the top ranked method. Within the Successors section, determine which codethis method called next.

k. Close the Flight Recording file view, but do not close JRMC.

Tip: For those with some Java knowledge, you can also view the culprit code at/resources.

5. Profile heap usage with the Memory Leak Detector.


58/128



Chapter 6 - Page 10

a. Execute the runadminclients.sh script.

b. While the test client executes, return to Mission Control. Launch the Console viewonce again for MedRecSvr1. Monitor server heap usage during the test:

Tip: For additional heap-related data, you can also use the Runtime > Memory tab.

c. After the test client finishes, close the console view.

d. Right-click the server in the JVM Browser view and select Start Memleak.

The memory leak view is displayed, which includes a table showing possible culpritJava types or classes. For example:

e. By default the table is sorted by Growth Rate. Instead click the % of Heap column tochange the sort criteria.

f. Select the top ranked type. Right-click and select List Largest Arrays.

The instances are displayed on the left.

g. Browse through the array instances in memory and their sizes.

Tip: For additional details about an object instance in memory, you can right-click itand select Inspect Instance. The Inspector view appears.

Tip: Once again, you can view the culprit code at /resources.

h. When finished with the practice, do the following:

1) Stop Mission Control.

2) Kill MedRecSvr1 and close its Linux terminal window.

3) Restart the server by using a new Linux terminal (to use the default environmentvariables).



/domains/MedRecDomain location does not yet exist, follow the SolutionInstructions for the “Developing a Custom Monitoring Script” practice.


59/128



Chapter 7 - Page 1


Chapter 7


60/128



Chapter 7 - Page 2


Practices Overview

In this practice you look into the Java classpath for possible errors.


61/128



Chapter 7 - Page 3

Practice 7-1: Investigating Classpath Problems


Skills Learned


• Locate and analyze a classpath error

•

Browse application and server classes• Debug server start scripts

Overview

Java code is made available to a JVM process, such as WebLogic Server, through classloaders. Class loaders are typically associated with some set of locations on the file system.The main JVM system class loader, for example, is configured by using the classpath argument. Applications also have their own class loaders. While the system classpath can be set to anyarbitrary file system paths and archives, WLS and the Java EE specification dictate from whereapplication-specific code can be loaded.

In this practice, the MedRec system has been upgraded to utilize a new third-party library. Whilethe application functions properly in the development environment, the production environment

has encountered a class loading error. You investigate the contents of the application along withthe system classpath.

Instructions

1. Set up the practice.


.


ant setup_exercise

The Lab Framework:

− Deploys a new version of the MedRec application

−

Updates the MedRec domain filesc. Kill and restart the server MedRecSvr1. Be sure to continue using the provided

startServer1.sh script.

2. Trigger a Class Not Found error.

a. Launch a Web browser and access the MedRec application:

http://localhost:7021/medrec/index.action

b. Log in as the patient Fred with: [email protected]/weblogic.

c. Click the Profile button.

d. Scroll down and click the Save button.

e. Confirm that the application fails and that you receive a generic HTTP 500 error.

f. Locate the Java error message in the log (or server output) for MedRecSvr1. It shouldresemble the following:


62/128



Chapter 7 - Page 4

javax.ejb.EJBException:

java.lang.NoClassDefFoundError:org/integratex/mainframe/XAdapter

at com.bea.medrec.service.impl.

PatientServiceImpl.updatePatient ...

...

at com.bea.medrec.web.controller.ViewingPatientController.updatePatient ...

Tip: The original exception was triggered from MedRec EJB application code, but thenwas caught by the Java Server Faces (JSF) framework in the Web application and

wrapped within a FacesException.

3. Investigate the application classpath.

After talking with the development team and searching the Internet, you have discovered

that the missing class, XAdapter, is a component of a third-party library. Most of theseresources also suggest that the class is typically found within an archive named

adapters.jar.

a. From the Linux File Browser, navigate to and open the/applications/medrec.ear file by using the Archive Manager.

b. Inspect the contents of the archive’s APP-INF/lib folder. Check for the presence ofthe file adapters.jar or a similarly named file.

c. Inspect the contents of the archive’s APP-INF/classes folder. Check for thepresence of the following folders and file:

org/integratex/mainframe/XAdapter.class.

d. Continue your search within the following archives found in medrec.ear. Browse theindicated locations if they exist:

File Location

medrec-domain.jar . (root)

medrec-web.war WEB-INF/classes

medrec-web.war WEB-INF/lib

Tip: You must extract the WAR file in order to view it with the Archive Manager.

Tip: You can also search archives with command-line tools such as jar. For example:

jar tvf medrec.ear | grep –i "adapter"

4. Investigate the server classpath.

a. Continue your search at /domains/MedRecDomain/lib.

Although your missing JAR file is not here either, you may notice an adapters.jar file in the domain’s root directory, MedRecDomain.

b. Locate the terminal running MedRecSvr1. Scroll up to the start script’s initial output.Locate the line at which the script prints the current classpath:

JAVA Memory arguments:...

...

WLS Start Mode=...


63/128



Chapter 7 - Page 5

...

CLASSPATH=...

Tip: If necessary, you can restart the server to view its initial output.

c. Scan this output for the presence of the adapters.jar file. Can you spot the error?

5. Correct the classpath issue.

a. Search the following start script files to try to locate the line at which adapter.jar isadded to the classpath:

MedRecDomain/bin/setDomainEnv.sh

MedRecDomain/bin/startWebLogic.sh

MedRecDomain/bin/startManagedWebLogic.sh

MedRecDomain/startServer1.sh

b. Remove this line from whichever script file in which you find it.

c. Move adapters.jar to the MedRecDomain/lib location.

Why? Any JAR file in the lib folder of a domain is automatically added to theclasspath of servers in the domain.

Tip: Alternative solutions include adding the JAR to the MedRec application ordeploying it as a shared library.

d. Restart the server by using a new Linux terminal. Confirm that the JAR is no longer inthe classpath.

e. Repeat the earlier steps to test the MedRec application again. The user’s profile shouldnow update with no errors:



/domains/MedRecDomain location does not yet exist, follow the SolutionInstructions for the “Developing a Custom Monitor

Oracle troubleshooting

Documents

Transcript of Oracle troubleshooting