Large-Scale Implementation of IBM Tivoli Composite Application Manager for WebSphere and Response...

7/31/2019 Large-Scale Implementation of IBM Tivoli Composite Application Manager for WebSphere and Response Time Tracking Redp4162

1/172

ibm.com/redbooks Redpaper

Large-Scale Implementation ofIBM Tivoli Composite ApplicationManager for WebSphere and

Response Time Tracking

Budi Darmawan

Aleem Subhedar

Celena Tan

Howard Anglin

Huang Chuan

Rohit Dhall

Planning for performance of

management infrastructure

Implementing with multiple

servers

Performing mass update

of agents

Front cover
http://www.redbooks.ibm.com/http://www.redbooks.ibm.com/http://www.redbooks.ibm.com/http://www.redbooks.ibm.com/


2/172


3/172

Large-Scale Implementation of IBM TivoliComposite Application Manager for WebSphereand Response Time Tracking

December 2007

International Technical Support Organization


4/172

Copyright International Business Machines Corporation 2006, 2007. All rights reserved.

Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADPSchedule Contract with IBM Corp.

Second Edition (December 2007)

This edition applies to Version V6.0 of ITCAM for Response Time Tracking (product number5698-A75) and Version 6.0 of ITCAM for WebSphere (product number 5698-A71).

Note: Before using this information and the product it supports, read the information inNotices on page vii.


5/172

Copyright IBM Corp. 2006, 2007. All rights reserved. iii

Contents

Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiTrademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixThe team that wrote this Redpaper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix

Become a published author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiComments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi

Chapter 1. Overview of IBM Tivoli Composite Application Manager

implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Application management with IBM Tivoli. . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.1.1 IBM Tivoli systems management portfolio . . . . . . . . . . . . . . . . . . . . . 3

1.1.2 IBM Tivoli Composite Application Manager solution . . . . . . . . . . . . . . 51.2 Scope of and concerns relating to large-scale implementation. . . . . . . . . . 6

1.2.1 Defining large-scale implementation. . . . . . . . . . . . . . . . . . . . . . . . . . 61.2.2 Concerns and considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.3 Overview of IBM Tivoli Composite Application Manager. . . . . . . . . . . . . . . 8

1.3.1 Understanding ITCAM for WebSphere . . . . . . . . . . . . . . . . . . . . . . . . 81.3.2 Understanding IBM Tivoli Composite Application Manager

for Response Time Tracking. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111.4 Document organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Chapter 2. Planning for ITCAM for WebSphere . . . . . . . . . . . . . . . . . . . . . 172.1 Planning considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182.2 Product architecture of ITCAM for WebSphere. . . . . . . . . . . . . . . . . . . . . 182.3 Deciding on the size of the servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

2.3.1 Sizing parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212.3.2 Sizing estimation for ITCAM for WebSphere managing server. . . . . 222.3.3 Data collector overhead . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

2.4 Implementation options for ITCAM for WebSphere. . . . . . . . . . . . . . . . . . 252.4.1 Designing the managing server . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252.4.2 Deploying a large number of data collectors. . . . . . . . . . . . . . . . . . . 28

2.5 Communication and security considerations. . . . . . . . . . . . . . . . . . . . . . . 292.5.1 Communication security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292.5.2 Firewall and port consideration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

2.6 Reliability and high availability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322.6.1 Failover and fault tolerance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322.6.2 Disaster recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32


6/172

iv Large-Scale Implementation of IBM Tivoli Composite Application Manager for WebSphere and Response Time Tracking

Chapter 3. Installing ITCAM for WebSphere . . . . . . . . . . . . . . . . . . . . . . . . 333.1 Installing ITCAM for WebSphere managing server. . . . . . . . . . . . . . . . . . 34

3.1.1 Installation configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

3.1.2 Database installation and configuration . . . . . . . . . . . . . . . . . . . . . . 35

3.1.3 WebSphere Application Server considerations. . . . . . . . . . . . . . . . . 413.1.4 Configuring the split server of the managing server . . . . . . . . . . . . . 423.1.5 Installation and setup of split server . . . . . . . . . . . . . . . . . . . . . . . . . 433.1.6 Verifying the installed components in a split environment . . . . . . . . 473.1.7 Adding additional publish servers and archive agents . . . . . . . . . . . 48

3.2 Deploying data collectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 503.2.1 Setting up the silent installation process . . . . . . . . . . . . . . . . . . . . . . 503.2.2 Installing the data collector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

3.2.3 Installing and configuring the data collector together . . . . . . . . . . . . 54

3.2.4 Configuring data collectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 563.2.5 Automatically discovering the installation parameters . . . . . . . . . . . 61

3.3 Configuring and setting up SSL communication . . . . . . . . . . . . . . . . . . . . 643.3.1 Managing server Secure Socket Layer setup . . . . . . . . . . . . . . . . . . 643.3.2 Data collector Secure Socket Layer setup . . . . . . . . . . . . . . . . . . . . 65

3.3.3 Working with custom certificates. . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

Chapter 4. Maintenance of ITCAM for WebSphere . . . . . . . . . . . . . . . . . . . 754.1 Operating ITCAM for WebSphere . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

4.2 Performance and availability of the managing server . . . . . . . . . . . . . . . . 764.2.1 Performance of the WebSphere Application Server . . . . . . . . . . . . . 764.2.2 Database maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 774.2.3 Data trimming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

4.3 Backup and recovery configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 834.3.1 ITCAM for WebSphere backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 834.3.2 WebSphere configuration backup. . . . . . . . . . . . . . . . . . . . . . . . . . . 834.3.3 Database backup and restore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

4.4 Log files and configuration files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 854.4.1 Managing log files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 854.4.2 Managing the configuration files . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

4.5 Performing product maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 874.5.1 Getting software updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

4.5.2 Updating ITCAM for WebSphere managing server. . . . . . . . . . . . . . 874.5.3 Updating ITCAM for WebSphere data collectors . . . . . . . . . . . . . . . 90

Chapter 5. Planning for ITCAM for Response Time Tracking . . . . . . . . . . 93

5.1 Planning considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 945.2 Product architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 945.3 Sizing the servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

5.3.1 Sizing parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96


7/172

Contents v

5.3.2 Sizing estimation for ITCAM for Response Time Tracking . . . . . . . . 975.4 Deployment of ITCAM for Response Time Tracking. . . . . . . . . . . . . . . . . 98

5.4.1 Designing the management server . . . . . . . . . . . . . . . . . . . . . . . . . . 985.4.2 Deploying the management agent . . . . . . . . . . . . . . . . . . . . . . . . . . 99

5.5 Communication and security considerations. . . . . . . . . . . . . . . . . . . . . . 1005.5.1 Communication security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1005.5.2 Firewall and port considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

5.6 Reliability and high availability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1025.6.1 Failover and fault tolerance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1025.6.2 Disaster recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

Chapter 6. Installing ITCAM for Response Time Tracking. . . . . . . . . . . . 1056.1 Clustering the management server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106

6.1.1 Preparing the operating system . . . . . . . . . . . . . . . . . . . . . . . . . . . 1066.1.2 Installing the database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1076.1.3 Installing WebSphere Application Server . . . . . . . . . . . . . . . . . . . . 1096.1.4 Installing WebSphere Load Balancer . . . . . . . . . . . . . . . . . . . . . . . 1186.1.5 Installing the management server . . . . . . . . . . . . . . . . . . . . . . . . . . 1246.1.6 Checking the configuration of the RTT cluster application . . . . . . . 128

6.2 Deploying the management resources . . . . . . . . . . . . . . . . . . . . . . . . . . 1296.2.1 Silent installation of the management agent . . . . . . . . . . . . . . . . . . 1296.2.2 Command-line interface for management components . . . . . . . . . 130

6.2.3 Defining management resources . . . . . . . . . . . . . . . . . . . . . . . . . . 1316.3 Setting up Secure Sockets Layer certificates . . . . . . . . . . . . . . . . . . . . . 133

6.3.1 Secure Sockets Layer for ITCAM for Response Time Tracking . . . 1336.3.2 Working with custom certificates. . . . . . . . . . . . . . . . . . . . . . . . . . . 134

Chapter 7. Maintenance of ITCAM for Response Time Tracking . . . . . . 1377.1 Operational issues pertaining to a large environment . . . . . . . . . . . . . . . 1387.2 Performance and availability of management server . . . . . . . . . . . . . . . 138

7.2.1 Performance of WebSphere Application Server . . . . . . . . . . . . . . . 138

7.2.2 Database maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1397.3 ITCAM for Response Time Tracking files . . . . . . . . . . . . . . . . . . . . . . . . 140

7.3.1 Backup and recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1407.3.2 Managing the log files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141

7.4 Performing product maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1427.4.1 Getting software updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1427.4.2 Updating ITCAM for Response Time Tracking management server1427.4.3 Updating ITCAM for Response Time Tracking management agents143

Abbreviations and acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145

Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147


8/172

vi Large-Scale Implementation of IBM Tivoli Composite Application Manager for WebSphere and Response Time Tracking

Other publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147Online resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148How to get IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149Help from IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151


9/172

Copyright IBM Corp. 2006, 2007. All rights reserved. vii

Notices

This information was developed for products and services offered in the U.S.A.

IBM may not offer the products, services, or features discussed in this document in other countries. Consultyour local IBM representative for information on the products and services currently available in your area.Any reference to an IBM product, program, or service is not intended to state or imply that only that IBMproduct, program, or service may be used. Any functionally equivalent product, program, or service thatdoes not infringe any IBM intellectual property right may be used instead. However, it is the user'sresponsibility to evaluate and verify the operation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matter described in this document.The furnishing of this document does not give you any license to these patents. You can send license

inquiries, in writing, to:IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY 10504-1785 U.S.A.

The following paragraph does not apply to the United Kingdom or any other country where such provisionsare inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDESTHIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED,INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT,MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimerof express or implied warranties in certain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes are periodically madeto the information herein; these changes will be incorporated in new editions of the publication. IBM may

make improvements and/or changes in the product(s) and/or the program(s) described in this publication atany time without notice.

Any references in this information to non-IBM Web sites are provided for convenience only and do not in anymanner serve as an endorsement of those Web sites. The materials at those Web sites are not part of thematerials for this IBM product and use of those Web sites is at your own risk.

IBM may use or distribute any of the information you supply in any way it believes appropriate withoutincurring any obligation to you.

Information concerning non-IBM products was obtained from the suppliers of those products, their publishedannouncements or other publicly available sources. IBM has not tested those products and cannot confirm

the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions onthe capabilities of non-IBM products should be addressed to the suppliers of those products.

This information contains examples of data and reports used in daily business operations. To illustrate themas completely as possible, the examples include the names of individuals, companies, brands, and products.All of these names are fictitious and any similarity to the names and addresses used by an actual businessenterprise is entirely coincidental.

COPYRIGHT LICENSE:This information contains sample application programs in source language, which illustrates programmingtechniques on various operating platforms. You may copy, modify, and distribute these sample programs inany form without payment to IBM, for the purposes of developing, using, marketing or distributing application

programs conforming to the application programming interface for the operating platform for which thesample programs are written. These examples have not been thoroughly tested under all conditions. IBM,therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy,modify, and distribute these sample programs in any form without payment to IBM for the purposes ofdeveloping, using, marketing, or distributing application programs conforming to IBM applicationprogramming interfaces.


10/172

viii Large-Scale Implementation of IBM Tivoli Composite Application Manager for WebSphere and Response Time Tracking

Trademarks

The following terms are trademarks of the International Business Machines Corporation in the United States,other countries, or both:

Redbooks (logo) pSeriesz/OSAIXCICSDatabase 2DB2 Universal DatabaseDB2ETE

ETEWatchIBMIMSLotus NotesLotusMonitoring On DemandMVSNotesOperating System/400

OMEGAMONOS/400RationalRedbooksTivoliWebSphereWorkplace

The following terms are trademarks of other companies:

Oracle, JD Edwards, PeopleSoft, Siebel, and TopLink are registered trademarks of Oracle Corporationand/or its affiliates.

Snapshot, and the Network Appliance logo are trademarks or registered trademarks of Network Appliance,Inc. in the U.S. and other countries.

ITIL is a registered trademark, and a registered community trademark of the Office of GovernmentCommerce, and is registered in the U.S. Patent and Trademark Office.

Enterprise JavaBeans, EJB, Java, JavaBeans, JDBC, JMX, JNI, JRE, JVM, J2EE, Solaris, and all

Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, orboth.

Microsoft, Outlook, Windows, and the Windows logo are trademarks of Microsoft Corporation in the UnitedStates, other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Linux is a trademark of Linus Torvalds in the United States, other countries, or both.

Other company, product, or service names may be trademarks or service marks of others.


11/172

Copyright IBM Corp. 2006, 2007. All rights reserved. ix

Preface

This IBM Redpaper discusses large-scale implementation of IBM TivoliComposite Application Manager for WebSphere and IBM Tivoli CompositeApplication Manager for Response Time Tracking. Large-scale implementation istypically characterized by the number of monitoring agents deployed and thenumber of transactions load-managed. A typical large-scale implementation of amonitoring product contains the following challenges:

Keeping up the performance of the monitoring tools to accommodate theprocessing load from the agents.

Automation of installation, update, and maintenance of monitoring agentsbased on silent installation and automated update.

Specific day-to-day maintenance actions to ensure performance andavailability of the monitoring solution.

This IBM Redpaper addresses these issues with regard to the implementation ofITCAM for WebSphere and ITCAM for Response Time Tracking on distributedplatforms. The discussion is divided into planning issues, implementation guides,

and maintenance considerations.

The team that wrote this Redpaper

This Redpaper was produced by a team of specialists from around the worldworking at the IBM International Technical Support Organization (ITSO), AustinCenter.

Budi Darmawan is a Consulting IT Specialist at the IBM ITSO, Austin Center.He writes extensively and teaches IBM classes worldwide on all areas of Tivolisystems management products. Before joining the ITSO in 1999, Budi worked asSolution Architect and Implementer in Integrated Technical Services, IBMIndonesia. His current interests include availability management, z/OS systemsmanagement, and Java programming.

Aleem Subhedar is a Staff Software Engineer with India Software Labs in Pune,India. He has seven years of experience in AIX and Middleware System

Administration. He holds a degree in Chemistry from Pune University. His areasof expertise include AIX, pSeries, and related system technologies. He is anIBM Certified System Expert. His areas of interest include pSeries virtualizationand high availability.


12/172

x Large-Scale Implementation of IBM Tivoli Composite Application Manager for WebSphere and Response Time Tracking

Celena Tan is a Managing Consultant with IBM Software Group Services inAustralia. She has 14 years of experience in the IT field. She holds a Masters ofTechnology from National University of Singapore and a Bachelor of ElectricalEngineering (Hons) from the University of Tasmania. Her areas of expertiseinclude ITCAM family products and rational testing, and change and

configuration management products.

Howard Anglin is a Deployment Expert for ITCAM for WebSphere, ResponseTime Tracking, IBM Tivoli Monitoring in the United States. He has worked withvarious large customers, and in his role as an IT Specialist he has resolveddeployment, integration, and performance issues. He has nine years ofexperience in the software test and development field with emphasis on theWebSphere Application Server. He holds a Bachelor of Science in ElectricalEngineering from Manhattan College, Riverdale, New York. Howard began his

career at IBM in the pSeries Hardware Group as a Test Engineer developingautomation solutions for the production line. He then transferred to the softwaregroup.

Huang Chuan is a Senior Test Lead of IBM China CSDL lab. He has five yearsof experience in software developing and over six years of experience insoftware product testing. He has led the ITCAM for Response Time Tracking testproject for several releases. He holds a degree in Computer Science from theUniversity of Electronic Science and Technology of China.

Rohit Dhall is an IT Architect with GBS, IBM India. He has 10 years of ITexperience in technologies like client-server computing, Web-basedtransactional systems, data warehousing, and data mining. His major expertise isin designing, implementing, and tuning large-scale Internet banking, eMortgage,and anti-money laundering solutions for the banking and financial sector. He isEXIN ITIL certified and also holds certification in Java and EJB fromBrainbench. His current interests include SOA and IBM Virtualization offerings.

Thanks to the following people for their contributions to this project:

Donna Martin, Noel Lewis, Tony Williams, Marco De Gregorio, Sushanto PanditIBM Software Group, Tivoli Software

John HortonAuthor of the first edition of Large-Scale Implementation of IBM Tivoli CompositeApplication Manager for WebSphere and Response Time TrackingLarge-Scale

Implementation of IBM Tivoli Composite Application Manager, REDP-4162

Julie CzubikInternational Technical Support Organization, Poughkeepsie Center


13/172

Preface xi

Become a published author

Join us for a two-week to six-week residency program! Help write an IBMRedbook dealing with specific products or solutions, while getting hands-onexperience with leading-edge technologies. You will team with IBM technicalprofessionals, Business Partners, or customers.

Your efforts will help increase product acceptance and customer satisfaction. Asa bonus, you will develop a network of contacts in IBM development labs, andincrease your productivity and marketability.

Find out more about the residency program, browse the residency index, andapply online at:

ibm.com/redbooks/residencies.html

Comments welcome

Your comments are important to us!

We want our papers to be as helpful as possible. Send us your comments aboutthis Redpaper or other Redbooks in one of the following ways:

Use the online Contact us review book form found at:

ibm.com/redbooks

Send your comments in an e-mail to:

[email protected]

Mail your comments to:

IBM Corporation, International Technical Support OrganizationDept. HYTD Mail Station P099

2455 South RoadPoughkeepsie, NY 12601-5400
http://www.redbooks.ibm.com/residencies.htmlhttp://www.redbooks.ibm.com/residencies.htmlhttp://www.redbooks.ibm.com/http://www.redbooks.ibm.com/http://www.redbooks.ibm.com/contacts.htmlhttp://www.redbooks.ibm.com/contacts.htmlhttp://www.redbooks.ibm.com/http://www.redbooks.ibm.com/http://www.redbooks.ibm.com/residencies.htmlhttp://www.redbooks.ibm.com/residencies.html


14/172

xii Large-Scale Implementation of IBM Tivoli Composite Application Manager for WebSphere and Response Time Tracking


15/172

Copyright IBM Corp. 2006, 2007. All rights reserved. 1

Chapter 1. Overview of IBM TivoliComposite ApplicationManager implementation

This chapter provides an overview of the large-scale implementation issues forIBM Tivoli Composite Application Manager. This chapter covers the followingtopics:

1.1, Application management with IBM Tivoli on page 2

1.2, Scope of and concerns relating to large-scale implementation on

page 6 1.3, Overview of IBM Tivoli Composite Application Manager on page 8

1.4, Document organization on page 15

1


16/172

2 Large-Scale Implementation of IBM Tivoli Composite Application Manager for WebSphere and Response Time Tracking

1.1 Application management with IBM Tivoli

Computer-based applications are the lifeblood of modern enterprises. Mostbusiness processes are driven by the so-called computer application that

promotes productivity, automates processing, and minimizes human errors.These applications enable business persons to focus on what must be done,instead of how to do it. However, as business processes rely more on theseapplications, the applications become critical to the business. The applicationsmust be available for the execution of the business processes.

Most applications evolved from centralized applications typically managed by theinformation technology (IT) department or mainframe-based applications, whereall the application layers are maintained from the central mainframe. Today,

applications tend to have multiple layers, often distributed across differentservers, different platforms, and different components. These applications arecalled composite applications. This complicates the management of applicationson matters such as operational settings, problem determination, andperformance management.

Applications as a business-critical entity must be available with adequateresponse time for users to perform their tasks. With application componentsspread throughout the enterprise, problem determination and performance

management are typically complicated. There is no clear path for finding whichcomponent faces the problem. Is it the database? A network problem? Theapplication server experiencing a bottleneck? A user machine stall? Sometimes,these components even belong to different organizations.

Figure 1-1 shows a typical composite application. This is used by multiple usersthrough the Internet and intranet. It consists of multiple application layers, eachwith its own abstraction level. Some of the applications have the original backend in the mainframe transactions.

Figure 1-1 Composite application


17/172

Chapter 1. Overview of IBM Tivoli Composite Application Manager implementation 3

Composite applications are regarded as the ultimate application managementchallenge, as they span different application servers that communicate with eachother. This architecture enables modular application development, wherechanges in a layer may not affect other layers, but introduces the complexity ofmultiple components.

This paper demonstrates how to implement the IBM Tivoli Composite ApplicationManager family of products in a large-scale environment. This chapter introducesIBM Tivoli product portfolio and how IBM Tivoli Composite Application Managerproduct fits.

1.1.1 IBM Tivoli systems management portfolio

IBM Tivoli product solutions are aligned towards an overall IBM IT ServiceManagement approach. Figure 1-2 shows the IBM IT Service Managementportfolio structure.

Figure 1-2 IBM IT Service Management

This approach provides Information Technology Infrastructure Library-alignedautomation work flows. Future offerings will provide an open standard-based andconfiguration management database-based solution, as well as a workflowengine.

IT Operational

Management Products

IT ServiceManagement Platform

IT Process

Management Products

Best Practices

Change and Configuration

Management Database

Server, Network& Device

Management

StorageManagement

SecurityManagement

BusinessApplication

Management

ServiceDelivery

& Support

ServiceDeployment

InformationManagement

BusinessResilience

IT CRM &Business

Management

IT Operational

Management Products


IT Process

Management Products

IT Operational

Management Products


IT Process

Management Products

Best Practices


Management Database


Management Database


Management

StorageManagement

SecurityManagement

BusinessApplication

Management


Management

StorageManagement

SecurityManagement

BusinessApplication

Management

ServiceDelivery

& Support

ServiceDeployment


BusinessResilience

IT CRM &Business

Management

ServiceDelivery

& Support

ServiceDeployment


BusinessResilience

IT CRM &Business

Management


18/172


The operational management pillar shown in Figure 1-2 on page 3 is divided intosoftware families. The availability solution addressed in business applicationmanagement and server, network, and device management can be viewed as anintegrated offering, as shown in Figure 1-3.

Figure 1-3 IBM Tivoli software portfolio

As shown in Figure 1-3, the Tivoli software portfolio is divided into the followingcomponents:

Resource monitoring

Measures and manages IT resource performance, including servers,databases, and middleware.

Composite application management

Monitors and manages an application and its components, and understands

applications from the availability standpoint. Event correlation and automation

Correlates and automates events or faults that are generated by resourcemonitoring, application monitoring, or both to provide a concise root-causeanalysis of failure in the environment.

Orchestration and provisioning

Provides the ability to deploy or redeploy servers or components asrequested on demand to fulfill processing requirements, if the necessityarises as indicated by the correlation engine.

Business Service Management

Event Correlation and Automation

Resource Monitoring

Orchestration and Provisioning

Composite Application Management

Security Storage


19/172


Business service management

Provides a high-level view of business status as reflected by its underlyingmonitoring components. The view is either in real time or based on aservice-level agreement.

1.1.2 IBM Tivoli Composite Application Manager solution

The IBM Tivoli Composite Application Manager family resides in the applicationmanagement pillar of the Tivoli software portfolio. The current applicationmanagement portfolio consists of the following products:

ITCAM for Response Time Tracking V6.1

ITCAM for Response Time V6.2

IBM Tivoli Composite Application Manager for service-oriented architecture(SOA) V6.1

ITCAM for WebSphere V6.1

ITCAM for J2EE V6.1

ITCAM for Web Resources V6.2

ITCAM for CICS Transactions V6.1

ITCAM for IMS Transactions V6.1

IBM Tivoli OMEGAMON XE for Messaging V6.0

Figure 1-4 shows the scope of composite application management.

Figure 1-4 Composite application management

Response TimeTracking

WBI messagingWeb Services calls

WebSphereperformance

CICS/IMStransaction


20/172


Manage the overall composite application from the following sides:

Get the user side of response time and availability with ITCAM for ResponseTime Tracking.

Get IBM WebSphere middleware performance and analyze in-depth resource

usage through ITCAM for WebSphere. Manage messaging from IBM WebSphere Business Integration MQ Series

using IBM Tivoli OMEGAMON XE for IBM WebSphere Business Integration.For more details, refer to Implementing IBM Tivoli OMEGAMON XE for

WebSphere Business Integration V1.1, SG24-6768.

Manage message flow in an SOA environment and collect metrics for Webservice calls using IBM Tivoli Composite Application Manager forservice-oriented architecture (SOA).

Provide the integration view with a mainframe-based, back-end applicationsuch as Information Management System (IMS) or Customer InformationControl System (CICS) using ITCAM for IMS Transactions or ITCAM forCICS Transactions.

1.2 Scope of and concerns relating to large-scaleimplementation

This paper discusses large-scale implementation of IBM Tivoli CompositeApplication Manager. It specifically provides information about theimplementation of ITCAM for WebSphere and ITCAM for Response TimeTracking in large-scale environments. The discussion is about large-scaleimplementation in distributed and mainframe environments, and includes thefollowing topics:

1.2.1, Defining large-scale implementation on page 6

1.2.2, Concerns and considerations on page 7

1.2.1 Defining large-scale implementation

There are several indications relating to large-scale implementation. Theseindications are based on the following factors:

The number of application servers to be monitored

Each application server must have an agent installed to be monitored and

managed. With the number of application servers ranging from hundreds tothousands, additional care must be taken to manage the deployment,maintenance, and processing of the managing server.


21/172


The transaction rates on application servers

The transaction rates contribute to the overhead of the monitoring system. Abalance of data collection and system health must be achieved. A largenumber of transactions potentially require larger management serverprocessing.

The number of network sites

The number of network sites typically corresponds to the potential bottlenecksbetween the sites. The bottlenecks may be from production data, monitoringdata, or a security requirement such as a firewall.

The requirement for high availability or fail over

This additional requirement, although not directly related to the scale, istypically a must for a large-scale implementation.

The existence of multiple managed spaces that a site must handle

Managed space is defined as a group of environments with a singlemanagement database and a set of management server processes. Differentmanaged spaces are usually used to separate the production anddevelopment environments. They are also used to prepare and test thechanges to the management environment.

1.2.2 Concerns and considerationsFollowing is a list of concerns and considerations that are specific to alarge-scale environment:

Server size

As this is a large-scale implementation, sizing the servers to manage theenvironments is critical. The placement, configuration, and specification of asingle server or multiple servers must be predetermined in order to avoidbottlenecks in processing. This sizing must also take into considerationspecial processing requirements such as debugging and troubleshooting anddata collection and recovery.

Deploying agents

The number of agents that must be deployed are enormous and prohibitive tobeing performed manually. Automated efforts must be included in the abilityto deploy and implement the agents automatically with minimal manualintervention. This must cover initial deployment, fix pack implementation, andmaintenance action.


22/172


Security

This includes confidentiality support and firewall support.

Confidentiality support secures information transfer between the agentsand the servers.

Firewallsupport allows the sites to be secured, with management actionstill flowing through in order to effectively manage the environment.

Reliability

Fail over and fault tolerance are critical to maintain while monitoringbusiness-critical applications. The reliability factor must be promptlyaddressed and ensured.

Maintenance

Changes do happen, as with deployment. These changes must be applied toboth the servers and the agents. Special consideration must be provided for alarge-scale implementation with changes on both the servers and the agents.While server consideration applies to preserving, monitoring, and datacollection with minimal downtime, agent consideration relates to automatingthe deployment process with minimal manual intervention and outage.

This paper deals with and addresses these concerns for ITCAM for ResponseTime Tracking and ITCAM for WebSphere implementations.

1.3 Overview of IBM Tivoli Composite ApplicationManager

This section explains the following topics:

1.3.1, Understanding ITCAM for WebSphere on page 8

1.3.2, Understanding IBM Tivoli Composite Application Manager forResponse Time Tracking on page 11

1.3.1 Understanding ITCAM for WebSphere

This section provides an overview of ITCAM for WebSphere. The discussionincludes the following topics:

Features and functions on page 9

Components on page 9 Platforms supported on page 10


23/172


For more information about ITCAM for WebSphere, visit the following Web site:

http://www.ibm.com/software/tivoli/products/composite-application-mgr-websphere/

Features and functionsITCAM for WebSphere helps increase the performance and availability ofbusiness-critical applications by providing facilities for real-time problemdetection, analysis, and repair. Correlation spanning Java 2 Platform, EnterpriseEdition (J2EE), Customer Information Control System, and InformationManagement System, and diagnostics at the method level pinpoint codeproblems to help resolve problems quickly and reduce support and operationscosts.

Todays business processes often depend on a number of complex applications.Although most businesses have traditional monitoring tools to manage individualresources at a high level, many lack an integrated solution to automaticallymonitor, analyze, and resolve problems at the service, transaction, application,and resource levels. As a result, operations and development may take a longtime to identify, isolate, and fix composite application problems.

ITCAM for WebSphere is an application management tool that helps maintainthe availability and performance of on demand applications. It helps you to

quickly pinpoint, in real time, the source of bottlenecks in application code, serverresources, and external system dependencies. This product also providesdetailed reports that you can use to enhance the performance of yourapplications. ITCAM for WebSphere provides in-depth, WebSphere-basedapplication performance analysis and a tracing facility.

ITCAM for WebSphere enables multiple levels of analysis to get a complete viewof the application, depending on the requirement. From production-levelmonitoring to detailed heap and method debugging, it digs into Structured QueryLanguage (SQL) performance analysis without the need for database monitors. Itprovides SQL information and information about calls that were made throughJava Database Connectivity (JDBC). ITCAM for WebSphere provides acomposite status correlation for transactions that use Customer InformationControl System and Information Management System as the back end.

ComponentsITCAM for WebSphere contains the following components:

Managing server

This acts as the central component that manages and administers the datacollectors. It stores that data in a relational database repository. A Web-based
http://www.ibm.com/software/tivoli/products/composite-application-mgr-websphere/http://www.ibm.com/software/tivoli/products/composite-application-mgr-websphere/http://www.ibm.com/software/tivoli/products/composite-application-mgr-websphere/


24/172


application is provided to show monitoring results. This interface is also calledthe visualization engine.

Data collector

This runs on the application server and collects performance information for

the managing server. Tivoli Enterprise Monitoring Agent

This collects information that shows the status of the WebSphere ApplicationServer and sends it to the Tivoli Enterprise Monitoring Server for display onthe Tivoli Enterprise Portal. The Tivoli Enterprise Monitoring Agent is installedon individual machines where data collectors reside. This component ismoved to IBM Tivoli Composite Application Manager for Web Resources inVersion 6.2.

Platforms supportedFor a complete platform coverage list, refer to the following Web site:

http://publib.boulder.ibm.com/tividd/td/ITCAMWAS/prereq60/en_US/HTML/itcam6.html

Table 1-1 provides an overview of the platforms supported for ITCAM forWebSphere V6.

Table 1-1 Platforms supported for ITCAM for WebSphere

Component Software

Managing server operatingsystem

IBM AIX V5.2 and V5.3 Solaris 8 and Solaris 9 (SPARC) Hewlett-Packard UNIX (HP-UX) 11i 1 Windows 200 Server or Advanced Server with

Service Pack 4 (SP4) Windows 2003 Server Standard Edition/Enterprise Edition

(SE/EE) Red Hat Enterprise Linux (RHEL) 3.0 and 4.0 SUSE Linux Enterprise Server (SLES) 8 and 9

Managing server database IBM DB2 V8.1 Fix Pack 6 (FP6) or IBM DB2 V8.2 Oracle 8i SE R3 8.1.7, Oracle 9i SE R2 9.2, Oracle 10g

Managing server WebSphere WebSphere Application Server V5.1.x or WebSphere ApplicationServer V6.x
http://publib.boulder.ibm.com/tividd/td/ITCAMWAS/prereq60/en_US/HTML/itcam6.htmlhttp://publib.boulder.ibm.com/tividd/td/ITCAMWAS/prereq60/en_US/HTML/itcam6.htmlhttp://publib.boulder.ibm.com/tividd/td/ITCAMWAS/prereq60/en_US/HTML/itcam6.html


25/172


1.3.2 Understanding IBM Tivoli Composite Application Managerfor Response Time Tracking

This section provides an overview of ITCAM for Response Time Tracking. Itdiscusses the following topics:

Features and functions on page 9 Components on page 12 Platforms supported on page 14

For more information about ITCAM for Response Time Tracking, visit thefollowing Web site:

http://www.ibm.com/software/tivoli/products/composite-application-mgr-rtt/

Features and functionsITCAM for Response Time Tracking proactively recognizes, isolates, andresolves transaction performance problems by using robotic and real-timetechniques. It is an end-to-end transaction management solution that monitorsuser response time and helps you to visualize the transactions path through yourapplication systems, including the response time contributions of each step.ITCAM for Response Time Tracking uses Application Response Measurement(ARM) technology to track the response time of a distributed application.

Data collector platform AIX V5.2 and V5.3 Solaris 8 and 9 SPARC HP-UX 11i 1

Windows 200 Server or Advanced Server with SP4 Windows 2003 Server SE/EE RHEL 3.0 and 4.0 SLES 8 and 9 Red Flag Advanced Server (RFAS) 4.0 and 4.1(xLinux) IBM Operating System/400 (OS/400) V5.2 and V5.3 IBM z/OS V1.4, V1.5, or V1.6

Customer Information ControlSystem

V2.2, V2.3, and V3.1

Information ManagementSystem

V7.1, V8.1, and V9.1

Component Software
http://www.ibm.com/software/tivoli/products/composite-application-mgr-rtt/http://www.ibm.com/software/tivoli/products/composite-application-mgr-rtt/


26/172


Todays business processes often depend on composite applications that spanWeb servers, J2EE application servers, integration middleware, and mainframesystems. Although most businesses have traditional monitoring tools to manageindividual resources, many lack an integrated solution to automatically monitor,analyze, and resolve user response time problems. As a result, it may take a

long time to identify, isolate, and fix distributed transaction performanceproblems.

ITCAM for Response Time Tracking enables you to follow the path of a usertransaction end-to-end across your business infrastructure. You can drill down toeach step the transaction takes as it travels across multiple systems, andmeasure how each component of a transaction contributes to the overallresponse time. The entire transaction analysis process is transparent tocustomers and application developers. It collects transaction performance

through robot and browser simulation, in-depth J2EE server instrumentation, andfeedback from Customer Information Control System and InformationManagement System.

ITCAM for Response Time Tracking feeds the Tivoli Enterprise MonitoringServer to provide a comprehensive performance management solution on TivoliEnterprise Portal. This enables the development of custom monitoringworkspaces for managing enterprise applications.

ComponentsITCAM for Response Time Tracking consists of the following components:

Management server

This acts as the central point of contact for ITCAM for Response TimeTracking. It consists of a WebSphere-based J2EE application that performsthe management and administrative functions. The management serverstores data in a central database repository.

Store and Forward Agent

This relays traffic to and from the management agents. Typically, the Storeand Forward agent is used in a firewall environment. It consolidates the portrequirements for the connectivity.

Management agent

This performs the monitoring function. Typically, it investigates theperformance of the distributed application, depending on the managementcomponents deployed on it. The components that you can deploy are:

Generic Windows workstation

This allows deployment of IBM Rational Robot to measure transactionperformance.


27/172


Client Application Tracker

This uses IBM ETEWatch scripts to collect performance information.Default monitoring is available for measuring IBM Lotus Notes andMicrosoft Outlook performance.

Synthetic Transaction Investigator (STI)This performs Web-based transactions and measures the resultingresponse time.

Quality of Service monitoring agent

This collects information about user performance by acting as reverseproxy between the user and the Web server.

JavaTM 2 Platform, Enterprise Edition(J2EE) monitoring agentThis instruments and collects performance information about J2EE-basedapplication servers such as WebSphere or WebLogic.

Web Response Monitor component

Rational Performance Tester

Tomcat and JBoss monitoring component

Generic Application Response Measurement (ARM) agent

This collects ARM events from a custom-instrumented application.

Tivoli Enterprise Monitoring Agent for Tivoli Enterprise Monitoring Server

This feeds data from the ITCAM for Response Time Tracking server todisplay on the Tivoli Enterprise Portal.


28/172


Platforms supportedFor a complete platform coverage list, visit the following Web site:

http://publib.boulder.ibm.com/tividd/td/ITCAMRTT/prereq60/en_US/HTML/Version60.html

Table 1-2 provides an overview of the platforms supported for ITCAM forResponse Time Tracking V6.0.

Table 1-2 Platforms supported for ITCAM for Response Time Tracking

Component Software level

Management server operatingsystem

Microsoft Windows 2000 Server with SP4 Windows 2000 Advanced with SP4 Windows 2003 Server SE or EE

IBM AIX V5.2 or V5.3 Solaris 9 or 10 HP-UX 11i 1 RHEL 3.0 or 4.0 SLES 8 or 9

Management server database Oracle 9i SE 9.2 IBM DB2 V8.1 ESE with FP3+ (required for WebSphere

Application Server V5.1.x) IBM DB2 V8.1 ESE with FP6a+ (required for WebSphere

Application Server V6.x) IBM DB2 V8.2

Management server WebSphere WebSphere Application Server V5.1.x or later versions WebSphere Application Server V6.0.1.x or later versions

Management agent platform Windows 2000 Professional, Server or Advanced Server withSP4

Windows 2003 Server SE or EE Windows XP Professional with SP1

IBM AIX V5.2 or V5.3 Solaris 9 or 10 HP-UX 11i RHEL 3.0 or 4.0 SLES 8 or 9 RFAS 4.0 or 4.1 (xLinux) z/OS V1.4, V1.5, or V1.6 OS/400 V5.2 or V5.3
http://publib.boulder.ibm.com/tividd/td/ITCAMRTT/prereq60/en_US/HTML/Version60.htmlhttp://publib.boulder.ibm.com/tividd/td/ITCAMRTT/prereq60/en_US/HTML/Version60.htmlhttp://publib.boulder.ibm.com/tividd/td/ITCAMRTT/prereq60/en_US/HTML/Version60.html


29/172


1.4 Document organization

This paper discusses the following topics:

Before the implementation

Chapter 2, Planning for ITCAM for WebSphere on page 17, and Chapter 5,Planning for ITCAM for Response Time Tracking on page 93, discuss theplanning and sizing considerations.

The implementation

Chapter 3, Installing ITCAM for WebSphere on page 33, and Chapter 6,Installing ITCAM for Response Time Tracking on page 105, discussadditional steps that are required, such as reliability and automationconsiderations.

After the implementation

Chapter 4, Maintenance of ITCAM for WebSphere on page 75, andChapter 7, Maintenance of ITCAM for Response Time Tracking onpage 137, discuss maintenance considerations and operational concernsrelating to a large-scale implementation.


30/172



31/172

Copyright IBM Corp. 2006, 2007. All rights reserved. 17

Chapter 2. Planning for ITCAM forWebSphere

This chapter provides information about areas that must be considered duringthe planning phase of IBM Tivoli Composite Application Manager implementationin a large environment. This chapter discusses the following topics:

2.1, Planning considerations on page 18 2.2, Product architecture of ITCAM for WebSphere on page 18 2.3, Deciding on the size of the servers on page 21 2.4, Implementation options for ITCAM for WebSphere on page 25 2.5, Communication and security considerations on page 29 2.6, Reliability and high availability on page 32

2


32/172


2.1 Planning considerations

This section discusses the following aspects pertaining to large-scaleimplementations (see also 1.2.2, Concerns and considerations on page 7):

Understanding the product architectureThis allows you to make the correct decisions. Section 2.2, Productarchitecture of ITCAM for WebSphere on page 18, describes the architecturefor ITCAM for WebSphere.

Sizing the servers

This is important to correctly acquire adequate servers and choose a soundsoftware configuration option. Section 2.3, Deciding on the size of theservers on page 21, describes one approach.

Understanding the servers configuration options and agent deployment

This is discussed for ITCAM for WebSphere in 2.4, Implementation optionsfor ITCAM for WebSphere on page 25.

Planning for communication security

This is a mandatory step for an enterprise with business-critical and sensitiveinformation in a transaction environment. Section 2.5, Communication andsecurity considerations on page 29, discusses confidentiality and firewall

requirements. Discussing reliability, failover, and disaster recovery issues

These are the other mandatory aspects pertaining to a critical businessprocess on a large enterprise. Section 2.6, Reliability and high availability onpage 32 discusses this.

2.2 Product architecture of ITCAM for WebSphere

This section discusses the product architecture of ITCAM for WebSphere. Thisunderstanding is critical to plan and decide about the server configuration andother implementation issues. See also IBM Tivoli Composite ApplicationManager V6.1 Family Installation, Configuration, and Basic Usage, SG24-7151.

ITCAM for WebSphere V6.0 evolved from WebSphere Studio ApplicationMonitor (WSAM) and IBM Tivoli OMEGAMON XE for WebSphere. ITCAM forWebSphere observes and reports on the health of Java 2 Platform, Enterprise

Edition-based applications. It tracks the progress of applications as they traversethrough Java 2 Platform, Enterprise Edition (J2EE) application servers,

iddl d t d t t d d t b ll t b k d t


33/172

Chapter 2. Planning for ITCAM for WebSphere 19

middleware adapters and transports, and database calls, to back-end systemssuch as Customer Information Control System (CICS) or InformationManagement System (IMS) to extract business data or to invoke mainframebusiness processes.

The tracking of applications produces request traces, where the events in arequests life are recorded and stored in a monitoring repository database.ITCAM for WebSphere captures the CPU and the elapsed internal times whenevents are called and exited, measuring as far down as the CPU consumed andthe elapsed internal times charged to individual methods in J2EE classes. Themethods or events taking the most time are marked as an applications parts thatdeserve attention for runtime improvement studies and code optimizations.

ITCAM for WebSphere does not require modification of any J2EE or mainframeapplication code. Java Virtual Machine Tool Interface (JVMTI) interfaces andprimitives, along with WebSphere Performance Management Interface (PMI) andz/OS System Measurement Facility (SMF) 120 records, are ITCAM forWebSpheres principal data sources. The monitoring data is collected andanalyzed to offer a wealth of information about the health of J2EE applicationsand their servers.

Many system-level performance metrics are collected and reported about J2EEapplication servers. The status of the servers and their resources, particularly atvital checkpoints such as CPU utilization, memory usage, and the status ofinternal components such as database connection pools, Java Virtual Machine(JVM) thread pools, Enterprise JavaBeans (EJB) usage, and requestprocessing statistics, are very important in locating real-time problems with J2EEapplications. ITCAM for WebSphere brings attention to these critical indicatorswith real-time, graphical displays of their values and their trends over a span oftime.

ITCAM for WebSphere is a distributed performance monitoring application forapplication servers. Its components are connected through IP network

communication. The central component of ITCAM for WebSphere, the managingserver, is its heart and brain. It collects and displays various performanceinformation from application servers.

The application servers run a component of ITCAM for WebSphere called datacollector, which is a collecting agent that runs in the application server andsends monitoring information to the management server. These data collectorsoperate independently of each other.

Figure 2 1 shows the overall architecture of ITCAM for WebSphere


34/172


Figure 2-1 shows the overall architecture of ITCAM for WebSphere.

Figure 2-1 ITCAM for WebSphere architecture

The application monitor comprises the following main parts:

Managing server

A managing server comprises several Java-based components that providethe environment to collect and present management data.

Data collector agent

A data collector agent runs on each monitored application server, whetherJ2EE, Customer Information Control System (CICS), or InformationManagement System (IMS), and communicates essential operational data tothe managing server. Unique sampling algorithms maintain low CPU andnetwork overhead, while providing application-specific performanceinformation.

Web Server

Application servers with

ITCAM for WebSphere

Data collectors

Browser interface

ITCAM

for WebSphere

Managing Server

Tivoli Enterprise

Monitoring Server

and

Tivoli Enterprise

Portal Server

I

2 3 Deciding on the size of the servers


35/172


2.3 Deciding on the size of the servers

The scale of the implementation must decide the size of the servers to be used.Sizing determines the hardware configuration and implementation considerationof the servers. This section discusses the following topics:

2.3.1, Sizing parameters on page 21

2.3.2, Sizing estimation for ITCAM for WebSphere managing server onpage 22

2.3.1 Sizing parameters

The following parameters must be considered before deciding on the size of theservers:

The number of data collectors for ITCAM for WebSphere

This value assumes that the application servers run a similar load profile. Ifthe application servers have several load profiles, consider them in differentgroups.

The transaction rate for application servers

The number of transactions executed for each minute, when multiplied withthe number of data collectors or monitoring agents, gives the total amount of

transaction information captured for a given period.

The complexity of a transaction

It is not easy to understand the complexity of a transaction. This requires amore subjective approach than transaction rate counting, which can beretrieved from the transaction data or the application log. The relativecomplexity of transactions is determined by the number of method calls pertransaction. Typically, the number of methods a complex transaction invokesis around four to six times that of a simple transaction.

There are some product-specific parameters that affect sizing considerations.These parameters are built to filter out unimportant or insignificant informationfrom the data that is collected. These parameters are:

Data collection filter Sampling rate Monitoring level Listening policy mask Instrumentation level

2 3 2 Sizing estimation for ITCAM for WebSphere managing server


36/172


2.3.2 Sizing estimation for ITCAM for WebSphere managing server

Specific to ITCAM for WebSphere, consider the following parameters for sizing:

Communication bandwidth Memory size

Processing requirement Database size

Communication bandwidth

Several communication traffic flows exist between the managing server and thedata collector. The communication traffic flows are:

Initial communication with the kernel to collect configuration information

This only happens in the initial connection when the data collector is started.This configuration information consists of sending the configuration andmanaging server Java archives.

Management information to modify data collection level, sampling interval, orlogging level from the kernel

This happens by request or when scheduled by Monitoring On Demand.The size of this communication is small and negligible.

Visualization engine requests for current active transactions

The impact of these requests depends on the following factors:

The transaction rate and the average transaction response time that makeup the average number of in-flight transactions

The number of concurrent Web console users who may request the

in-flight transaction information

Transaction information is streamed to the publish server as it happens

This is the largest contributor to network load. It uses up the largest amount ofnetwork bandwidth. The formula is as follows:

Monitoring in level 1: transaction rate x 353

Monitoring in level 3: (transaction rate x 353) + (transaction rate x methodcall x 172)

Important: Sizing estimation for ITCAM for WebSphere managing servermust be estimated for a worst-case scenario, that is, in the state that level 3monitoring is run for the highest number of data collectors concurrently.

As an illustration, we use a sample environment with a transaction rate of


37/172


, p3,000 requests per minute on level 1 and 300 requests per minute on level 3monitoring. The average method calls is 500 methods per requests. Thetransaction bandwidth required is:

Level 1 transaction load: (3000 transaction / 60 seconds) x 353 = 17,650

bytes/sec

Level 3 transaction load: (30 transaction / 60 seconds) x 353 + (30 / 60) x5,000 x 172 = 430,176 bytes/sec

As shown in this example, the majority of network usage is spent on level 3analysis. In a real production environment, for the majority of time, ITCAM forWebSphere runs on level 1. Therefore, the communication requirement islow. However, prepare an installation to occasionally increase monitoring inlevel 3 for problem determination purposes.

Memory sizeMemory requirement is typically important for the following components:

Kernel

The memory size of the kernel is directly related to the number of datacollectors. The typical size of 64 MB in the setenv.sh may have to beincreased for more than 50 data collectors.

Publish serverThe memory size is related to the number of transactions the publish serverhas to process, with some consideration to the transaction complexity factor,that is, the number of methods invoked. The publish servers memory must beadequate to handle the data size between garbage collector intervals. Forgarbage collection per minute, you must accommodate a minutes worth ofdata. In the example provided in Communication bandwidth on page 22, thetotal size of publish server memory for processing the load must be around4.3 x 60 x (1.5) = 387 MB. Note that the base publish server was already

using around 100 MB of storage.

Archive agent

This requires memory as a subset to the publish server and is masked by thesampling percentage from the publish server. The archive agent uses morememory than the sampling rate percentage, as it performs Java DatabaseConnectivity (JDBC) database calls.

Visualization engine memory size


38/172


g y

This depends on the number of users who are connected and the activitiesthat they perform. Users are categorized into the following groups:

Users monitoring the availability screens

Users collecting performance reports Users monitoring in-flight threads

Modify the visualization engines memory size by using the WebSphereApplication Server administration console.

Memory sizes for ITCAM for WebSphere components are defined in thesetenv.sh file that is sourced by all overseer components.

Processing requirement

The processor requirement for ITCAM for WebSphere is directly related to thetransaction rate. The largest processor usage is for the following components:

Publish server: to process transaction data Database engine: for interface to the database Archive agent: to perform SQL calls WebSphere Application Server: to process user requests

Database size

The typical database size requirement depends on:

The number of application server statistics The transaction volume to be stored The complexity of transaction The duration to keep the data

Database table information that increases in size during ITCAM for WebSphereexecution is:

requests: number of requests x 353 bytes methods: number of methods x # requests in L3 x 172 bytes

pmidata: number of data collectors x (3600/polling interval) x 73 bytes

serverstats: number of data collectors x (3600/polling interval) x 107 bytes

volumestats: number of data collectors x (3600/polling interval) x 74 bytes

memorydata: number of data collectors x (3600/polling interval) x 115 bytes

gcdata: number of data collectors x (3600/garbage collection interval) x104bytes

2.3.3 Data collector overhead


39/172


Monitoring with ITCAM for WebSphere has overhead related to data collectorsrunning on a production WebSphere Application Server. The overhead isminimal for data collectors running on level 1 monitoring. This is typically around

a 23% increase of CPU time with no notable memory or disk input/output (I/O)requirement.

When the monitoring level is increased, the processing overhead of ITCAM forWebSphere data collectors also increases. This increase is due to the fact thatITCAM for WebSphere collects more data from more sources. A typical level 2monitoring generates around a 10% increase in processing usage, while a level3 monitoring generates around 2530% overhead.

This means that level 2 or level 3 monitoring must be used sparingly in your

production environment. To change the monitoring level for purposes of problemdetermination, schedule it to start and then step back to level 1 automatically inorder to reduce the impact on users.

2.4 Implementation options for ITCAM for WebSphere

Depending on the size of your implementation, there are some considerations for

implementing ITCAM for WebSphere. This section discusses the followingtopics:

2.4.1, Designing the managing server on page 25 2.4.2, Deploying a large number of data collectors on page 28

2.4.1 Designing the managing server

The ITCAM for WebSphere managing server consists of the following products:

IBM DB2 Universal Database Enterprise Server or Oracle database server WebSphere Application Server ITCAM for WebSphere managing server

Figure 2-2 shows the conceptual relationship between the components.


40/172


Figure 2-2 ITCAM for WebSphere components

The following ITCAM for WebSphere components are displayed in Figure 2-2:

Kernels

These control the managing server. There are always two copies of kernelsrunning on an ITCAM for WebSphere managing server for redundancy andfailover. The kernels register components as they join the managing server,periodically renew connections and registrations with components and datacollectors, and collect server and component availability information.

Publish servers

These receive application and system event data from the data collectors,

gather and compute request-level information about performance metricssuch as response times, and implement the trap monitoring and alertsfeatures.

Archive agents

These receive monitoring data from the publish servers and store themonitoring data in ITCAM for WebSpheres repository.

Global publishing server

This collects information from the publish servers and correlates all parts andpieces of multi-server requests, such as requests from J2EE servers toexecute Customer Information Control System (CICS) or InformationManagement System (IMS) programs.

Kernel (KL)Provide services on:

- Lookup- Registration- Recovery- Configuration

Publish Server (PS)

Archive Agent (AA)

Global Publish

Server (SAM)

Polling Agent (PA)

Message Dispatcher

(MD)

Visualization EngineProvide services on:-Administration-Availability-Problem Determination-Performance Management

OCTIGATEdatabase

P

ublish

traffic

Sn

apshottraffic

Message dispatcher


41/172


This is a conduit for messages from ITCAM for WebSphere using e-mail andSimple Network Management Protocol (SNMP) facilities.

Polling agent

This collects data from Web servers for Apache 2.0 and later versions. Visualization engine

This is a Web-based graphical user interface (GUI) with access to graphics,ITCAM for WebSphere performance reports, real-time views of different slicesof monitoring data, ITCAM for WebSphere internal commands, andevent-driven functions. The visualization engine runs on a J2EE server suchas WebSphere Application Server.

Although ITCAM for WebSphere provides the facility to install all the componentsin a single wizard, which is called embedded installation, individually installingeach component allows more flexibility in terms of verifying each component andconfiguring them to suit your requirements. The considerations that you mustkeep in mind when installing the components are:

Database

You can install the database locally on the managing server or on a separatedatabase server. ITCAM for WebSphere provides database configurationscripts to assist with the configuration of a remote database.

Utilizing a remote database, regardless of whether it is a DB2 UniversalDatabase or an Oracle database, relieves the processing load on themanaging server. An environment with hundreds of data collectors generatesa large amount of data flowing into the database. This amount increasesconsiderably if the data collectors are set to run monitoring at level 2 or level3, even for a short period of time.

A remote database allows database query processing and recording to beprocessed using dedicated hardware, instead of sharing with the mainmanaging server that is already busy with processing the transactioninformation.

WebSphere Application Server

The visualization engine of the managing server acts as the administrationconsole for ITCAM for WebSphere. The visualization engine is deployed on aWebSphere Application Server JVM that resides in a standalone applicationserver or an application server that is a part of a network deploymentenvironment.

We recommend that you install the visualization engine on a separateapplication server JVM that is not monitored by ITCAM for WebSphere datacollectors, especially in a network deployment environment. This reduces any

possible conflicts that may arise with respect to ITCAM for WebSphere. Inaddition if an issue does arise problem determination will be somewhat


42/172


addition, if an issue does arise, problem determination will be somewhateasier due to the separation.

ITCAM for WebSphere components

Configure the managing server to handle large amounts of data by addingadditional components, such as the publish servers and the archive agents.When adding the publish servers and the archive agents, the distribution ofdata is handled by the managing server. The amount of data being written tothe database is handled more efficiently as well.

Another major consideration for the managing server is the split serverinstallation. This option provides the managing server with the overseerprocesses that exist on separate machines, including the kernel, whichprovides load balancing and failover capabilities.

There are benefits to this type of configuration when there are hundreds ofdata collectors providing data to the managing server. This type of setup notonly allows the managing server to handle more memory and disk spaceusage, but also provides a failover capability. For more information about splitserver installation, refer to 3.1, Installing ITCAM for WebSphere managingserver on page 34.

2.4.2 Deploying a large number of data collectors

Installation of a small amount of ITCAM for WebSphere data collectors isperformed by using the graphical-based installation and configuration wizardprovided by the product. When presented with the task of deploying hundreds ofdata collectors into an environment, the graphical interface is no longer a goodoption. This non-interactive automated installation method is commonly knownassilent installation.

The use of silent installation provides a means to deploy a larger number of data

collectors in a more efficient manner. When performing the silent installation,information about the WebSphere environment must be known ahead of time. Aresponse file will be used during the installation, and if incorrect information isused, may result in a failed install.

When performing the silent installation of data collectors, the WebSphereApplication Server version must be taken into account, as V6 introduced theusage of profiles. The response files for silent installation are different for variousversions of the WebSphere Application Server. In some cases, when two

versions of WebSphere Application Server are present, it is better to have twoseparate master response files.

Installing the Tivoli Enterprise Monitoring Agent (TEMA) is also an option of thesilent installation Although Tivoli Enterprise Monitoring Agent can be installed


43/172


silent installation. Although Tivoli Enterprise Monitoring Agent can be installedusing silent installation, more configuration must be performed to connect to IBMTivoli Monitoring V6.1 Tivoli Enterprise Monitoring Server.

Mass automated installation is also possible by using a software distribution orprovisioning solution such as the IBM Tivoli Configuration Manager.

There is an additional consideration for deploying data collectors on a machinethat has multiple application servers installed. Consider installing a separate datacollector directory set for each application server, because applying a fix pack fordata collectors requires you to stop the application server. You have a moreflexible scheduling option with separate data collector installation for eachapplication server.

2.5 Communication and security considerations

Communication and security issues are vital to the inter-networked world that welive in. Applications and their management infrastructure must be secured inorder to protect resources from unauthorized sources. This section discusses thefollowing planning considerations:

2.5.1, Communication security on page 29 2.5.2, Firewall and port consideration on page 30

2.5.1 Communication security

Communication security relates to the confidentiality of the informationtransmitted over a network. Management information that is used by IBM TivoliComposite Application Manager products may contain details about applicationprocessing internals. This requires the content of the management information to

be secured from being accessed by unauthorized sources.

WebSphere securityWebSphere security plays a significant role in a large-scale implementation. Insome cases, WebSphere security is not enabled during the test phase of animplementation, but in a production environment. This requires certain additionalconsiderations. The WebSphere user must have the appropriate permissions to,for instance, issue awsadmin command.

The configuration of data collectors involves the use of Java CommandLanguage (JACL) scripts, and can fail when there is a permission problem.

If any of the application servers on which the data collector is installed hasWebSphere security enabled on it, the entire ITCAM for WebSphere


44/172


WebSphere security enabled on it, the entire ITCAM for WebSphereenvironment must have it enabled as well. This includes WebSphere securitybeing enabled on the ITCAM for WebSphere managing server.

Secure Sockets Layer communicationSecure communication between the managing server and the data collector is aviable option if there is a requirement for data to be encrypted duringtransmission. Using Secure Sockets Layer (SSL) provides secure datatransmission from the data collector to the managing server and must appeasecorporate security requirements, if necessary. Additional configuration must takeplace on the managing server and the data collector when enabling SSL. Acertificate key generator is included with the product. This key generatorprovides the facility to use custom-generated keys.

A best practice is to complete the default installation of the managing server andthe data collector and then enable SSL for both. This isolates problems (that is,whether the problem is caused by the basic installation or the SSL configuration).

2.5.2 Firewall and port consideration

Firewall and port issues arise when the data collectors are on a different site,location, or subnet from the managing server. Problems such as name resolution

occur if the Domain Name System (DNS) is not set up correctly on either themanaging server or the data collectors. Routing problems occur if the InternetProtocol (IP) addresses used belong to different subnets. The entire networkenvironment must be looked into in order to determine where a firewall, router, orbridge may exist.

Figure 2-3 shows the communication port requirement for ITCAM forWebSphere.


45/172


p

Figure 2-3 Communication port requirements

The managing server requires open ports for each kernel and publish server.The data collector requires open ports for the command agent and the eventagent. The port consolidator requires a port to communicate to the managingserver. Use a single port consolidator to consolidate communication frommultiple data collectors.

A port consolidator is useful to limit the number of ports required forcommunication between the data collector and the managing server. Portconsolidation is a viable option if there is a limit to the number of ports that canbe opened on the firewall. Additional configuration must be carried out on thedata collector, including the configuration of the data collector to go through theport consolidator, and starting the port consolidator process.

PortConsolidator

DC

Command Agent

DCEvent Agent

KL1

KL2

PS1

PS2

DCCommand Agent

DCEvent Agent

DCCommand Agent

DCEvent Agent

2.6 Reliability and high availability


46/172


This section discusses reliability issues that relate to failover and disasterrecovery.

2.6.1 Failover and fault tolerance

Split server configuration for ITCAM for WebSphere or the clustering server forITCAM for Response Time Tracking consists of having two or more managementservers running on separate physical machines. Hardware or software errors dooccur on a machine and cause the server to cease functioning. Using theseparate server configuration, the secondary server can handle the entire loaduntil the failing machine is recovered.

The switchover to the secondary managing server is not automatic. Manualintervention must take place for the failover to be successful. There are specificITCAM for WebSphere components that can only be run on one managingserver. They must therefore be started on a secondary server, such as the globalpublish server or the message dispatcher, if the primary server goes down.

2.6.2 Disaster recovery

There are three areas where a backup is necessary for disaster recovery with

Large-Scale Implementation of IBM Tivoli Composite Application Manager for WebSphere and Response...

Documents

Transcript of Large-Scale Implementation of IBM Tivoli Composite Application Manager for WebSphere and Response...