Generated by Jive on 2013-02-21+01001

Tom Cenens Beyond SAP Blog CornerSAPADMIN Root Cause Analysis statusupdate

Posted by Tom Cenens Mar 21 2011

Introduction

This blog is part of the SAPADMIN launch an idea that was launched to havea space for system administrators on SCN You can read the leading blog hereMartin English made a dummy concept page which you can check out Alsocheck todayrsquos blogs for hash tag SAPADMIN and check twitter for tweets thatare tagged with SAPADMIN You can find the idea on idea place here

Picture 11

I have been working on the root cause analysis some time now and while I love the productI still see a lot of room for improvements I thought it would be nice to share some tips tricksand write about the possible improvements

The tools surely has its benefits tracing HTTP content (Portal Web Dynpro) toidentify the component where the bottleneck lies gives can give nice results likepicture 11 where there was a network issue at the time I performed the trace Ifthe tools would be not be useful I wouldnrsquot frequently blog about it

Tom Cenens Beyond SAP Blog Corner SAPADMIN Root Cause Analysis status update

Generated by Jive on 2013-02-21+01002

Diagnostics Agent

The Diagnostics Agent is used in the root cause analysis scenario (also called diagnostics)In short the agent fetches data and sends it to the SAP Solution Manager where it is storedin BI Infocubes

Installation

The installation of diagnostics agent 720 is performed using a SAR file package whichfeatures a SAPinst once unpacked that you can use to install your diagnostics agent Youalso need the JCE Policy 6 zip file as Diagnostics Agent 720 is using SAPJVM 61

Host Agent included

The Host Agent is used to provide services that can be accessed by the SAP systemsresiding on the same host For diagnostics the main service provided is SAPOscol (SAPOS Collector) which collects OS statistics (CPU Memory Disk etc) The Host Agent is nowautomatically included in the SAPinst provided to install Diagnostics Agent 720

Mini SAP system

The Diagnostics Agent consists out of a mix of different processes that existed already Youcan see it as a mini SAP system Where in the past you had so use scripts to stopstart theagent you can simply use the standard commands startsap and stopsap now The build ofthe agent also looks very similar to a regular SAP system with profiles kernel and so on

The agent has a toned down SAP kernel present which consists out of a number ofprocesses that are part of the package SAPEXE Besides that you also have a SAPJVM 61package as the Diagnostics Agent is running on Java technology

Update update update

Some time ago a blog was posted by Karol Frankiewicz with information how to patch theSAPEXE to avoid errors upon startup What happened is that at startup the profiles arechecked and this would throw messages that there are missing parameters as the startupprocess thinks that the Diagnostics Agent is a SAP system which has an ICM on board Toget rid of these errors you can update the kernel files that are used by the agent

I mailed Karol and told him the instructions are not all that clear and I was happy to seehe responded very quickly and stated he would change the instructions based on ourconversations through mailing So you can expect to see more clear instructions on the

Tom Cenens Beyond SAP Blog Corner SAPADMIN Root Cause Analysis status update

Generated by Jive on 2013-02-21+01003

kernel updating of the agent in SAP Note 1466109 - Support of systemtype=SMDA in theSAP profiles

The kernel isnrsquot the only package you should update the SAPJVM 61 and theHost Agent should also be updated but more about that later in this blog

Since I do have other points which I wanted to address I decided to write this blog and givefeedback to the SCN community members to improve the installation process in order toavoid as much issues as possible afterwards

Pain points

There are definitely some pain points performing the installation using the regular SARfile provided During the installation you can specify a kernel package and a SAPJVM61package that you have downloaded This means the installer will then use these files insteadof the standard files that are delivered with the installation DVD

There goes the free disk space

Donrsquot be surprised if you run out of disk space as the SAPEXE package thatis delivered with the agent only holds around 78 files and the SAPEXE youdownload from marketplace (mentioned in the SAP Note to update theDiagnostics Agent kernel) holds at least 150 files

Another problem is that the SAPJVM 61 package you downloaded will beinstalled along with the initial version So you will have a SAPJVM 61001 inyour file system sitting and doing nothing and a SAPJVM 61ltversion youdownloaded) which is used I really donrsquot want that SAPJVM 61001 consumingdisk space

Shut it down

The Host Agent install is done automatically but it also means you run into problems if youinstall a second Diagnostics Agent 720 on the same host as your Host Agent processwill be running and cannot be overwritten by the automatic installation so you will have totemporarily shut it down

This also means that the Host Agent files get overwritten in the process sohopefully you didnrsquot update it manually yet because you will have to update itagain in that case Another possibility is of course that you edit the XML file andskip this particular step You can read more on editing XML files for SAPinst in oneof my previous blogs

Tom Cenens Beyond SAP Blog Corner SAPADMIN Root Cause Analysis status update

Generated by Jive on 2013-02-21+01004

How can these issues be solved

The proper solution is most likely to rewrite the SAPinst procedure for theDiagnostics Agent There should be a check if the Host Agent is already installedand if yes give the option to overwrite or keep the current version The SAPJVM61 should not be safeguarded in my opinion as itrsquos an initial installation and theold version will only consume disk space for no good reason The kernel files thatare necessary should be copied out of the provided SAPEXE SAR and not all thekernel files

The only good solution at this moment I see to avoid as much of these pain points aspossible is to remodel the installation DVD with the latest available version of thosecomponents In order to do that you need to find the files of course

Remodeling the installation DVD

Where are the damn files at

It can be a challenge sometimes to find files on SAP Service Marketplace Irsquomsure many can relate

SAPVJM

To be able to update you need to find the files first of course It seems there weremany questions on where to find the files because SAP Note 1442124 - How todownload a SAP JVM patch from the SMP states where to find the SAPJVM filesDonrsquot forget the V=MAINT at the end of the URL as it doesnrsquot come along if youcopy paste the URL from the SAP note

The instructions to manually update SAPJVM can be found in SAP Note 1025085 - How tomanually patch the SAPJVM

SAP Host Agent

You can find the SAP Host Agent by following the path on httpsservicesapcomswdc -gt Support Packages and Patches -gt Browse Our Download Catalog -gt SAP TechnologyComponents -gt SAP Host Agent

Updating the Host Agent is easy and goes quickly You can find instructions inone of my previous blogs The instructions to update the Host Agent are still thesame just make sure you use the right file version of course If you bump intothe issue that a file cannot be copied copy it manually from your temporary

Tom Cenens Beyond SAP Blog Corner SAPADMIN Root Cause Analysis status update

Generated by Jive on 2013-02-21+01005

directory into usrsaphostctrlexe and rerun the saphostexec ndashupgrade in yourtemporary directory to see it finish successfully

SAP Kernel

You can find the path to the actual SAP Kernel by clicking on the correct version under theSP Patch Level of SAP Note 1466109 - Support of systemtype=SMDA in the SAP profileswhich was created to address the warning messages during the startup when using an olderversion of the SAP Kernel for the Diagnostics Agent

Remodeling

When you unpack the SAR file delivered for the installation of the DiagnosticsAgent 720 you will see two DVDrsquos one which holds SAPinst and one which holdsthe packages of SAPJVM the Host Agent and the SAP Kernel

You can remodel the installation DVD by changingreplacing the SAR files contained in theKernel installation DVD

To change the Kernel files unpack the SAPJ2EESAR to a temporary folder X unpack thenew SAPEXE kernel to a temporary folder Y Create a list of files that are contained in folderX and launch a little script that loops through the list and only copies the files from folder Y tofolder X that are contained in the list That way you are sure you have all the necessary filesand as much as possible is updated with the latest kernel version

An alternative described in the SAP note (unpacking into global syncing global to localby starting the agent stopping the agent cleanup global exe and copy local exe to globalexe) but this requires you to have enough space available to perform these operationsGlobal exe dir is usrsapltSIDgtSYSexeucltplatformgt and local exe dir usrsapltSIDgtSMDAltinstance nogtexe

The SAPJVM6SAR files is easier download the latest SAPJVM 61 version and renamethe SAR package to SAPJVM6SAR and replace the original SAPJVM6SAR of the kernelinstallation DVD

The same goes for the SAPHOSTAGENTSAR file download the latest available versionrename and replace the original SAPHOSTAGENTSAR of the kernel installation DVD

SAPOscol errors in the Agent log files

The Diagnostics Agent 720 looks for the process saposcol in usrsapltSIDgtSMDAltinstance numbergtexe and if itrsquos not found throws an error in the

Tom Cenens Beyond SAP Blog Corner SAPADMIN Root Cause Analysis status update

Generated by Jive on 2013-02-21+01006

diagnostics agent log file While it would not be a problem as the Host Agent isrunning SAPOscol as a service and it should be accessible and you should alsosee data appearing in Wily Introscope you can get rid of the errors by copyingthe saposcol that is located in usrsaphostctrlexe to the path usrsapltSIDgtSMDAltinstance numbergtexe

If tenured space usage is not showing create a customer message

The troubleshooting guide which is available on SAP Service Marketplace hassome interesting content but in my opinion more should be added to the guideSAP support had many messages according to what I have seen and they shouldalready have an extensive knowledge base on possible problems Surely theyhave content on what to check if the tenured space usage isnrsquot showing in WilyIntroscope It would be a good idea to update the troubleshooting guide with asmuch information as possible to enable to customer to do more initial searchingand possibly solving the issue without the need to open a customer message

In my opinion customers implement the scenario and often donrsquot really knowor realize something is missing since there is no alerting that says ldquohey you aremissing somethingrdquo they assume everything works as it should be but in factoften there are metrics missing

Pain points for the implementation of the root cause scenario

I have been working with root cause analysis some time now in the beginningthere was a lot of frustration today there still is some frustration but slowly itrsquosgetting better bit by bit There are still some major paint points when you haveto implement this at a customer site If the prerequisites are not fully met expectto have a 90 working scenario or even less

While you could say of course you have to meet the prerequisites itrsquos not alwaysfeasible in a short amount of time The problem starts with the correctness of thesystem landscape in transaction SMSY in SAP Solution manager

In general the transaction is poorly maintained by most of the customers do I blame thecustomers then Not at all the way the transaction is build and the way the data is updatedis not good enough

The recommended way is to have the managed SAP system push data to the SLD ofthe Solution Manager Directly or through an SLD bridge That data is then fetched in theSolution Manager from the SLD to fill transaction SMSY However if you have ever set thedata fetching to use method RFCTMS the data is not updated automatically when the fetch

Tom Cenens Beyond SAP Blog Corner SAPADMIN Root Cause Analysis status update

Generated by Jive on 2013-02-21+01007

takes place (paint point right here) Clicking on a button to read the system number of themanaged SAP system is enough to change the fetching method from SLD to RFCTMSChanging it back is not possible you would have to delete the SAP system and recreate itthrough a SLD data fetch

The job that fetched that data titled LANDSCAPE FETCH which you can find intransaction SM37 can have seriously long runtimes Two thousand seconds isvalue I have seen frequently You can expect to see a change in this mechanismin Solution Manager 71 though but I donrsquot know to what extent they will actuallyfix the other problems

The only way to get the data corrected properly is to delete the managed SAP system intransaction SMSY along with the data that belongs to the SAP system (database javainstance) Again pain points here as the SAP system might be included in numerous SAPSolution Manager projects which are then again used by other scenarios such as CHARM(Change Request Management)

Fixing the content would then mean removing the SAP system out of everyscenario that is in use Are you serious Yes It doesnrsquot stop there because ifthe product assignment of the SAP system is incorrect you are soon looking atchanging the product assignment of all the SAP systems contained in the logicalcomponent in order to switch the logical component to the correct product Takeinto account the fact that you have to wait for two-thousand seconds when youperform the change

How do you get rid of inactive old agents

Picture 12

Tom Cenens Beyond SAP Blog Corner SAPADMIN Root Cause Analysis status update

Generated by Jive on 2013-02-21+01008

You have two different locations were your old agents might reside One is among theoffline agents in the agent administration application (accessible through root cause analysisworkcenter) You can see in picture 12 there 14 agent(s) offline Some or all of them areobsolete old and can be removed

Picture 13

To remove the offline agents which are no longer running or required first clickthe show offline agentsrsquo button to show in the list of agents Next click on theFilter on button and type a filter to have one or more agents remaining in the list(the agents you wish to remove) Next hit the Remove offline agent entries (seepicture 13)

Agent Candidate management

Picture 14

There is another location in the agent administration application were you could haveobsolete entries The Agent Candidates Management (see picture 14)

Picture 15

Click on the Agent Candidates Management link and next choose custom SLD ConnectionParameters (see picture 15) Maintain the parameters listed below and click the Applybutton You will then see the agents that are connected to the SLD which were once insertedinto the Agent Candidates Newer agents register directly if you keep default settings duringinstallation and they will not appear in the list

Tom Cenens Beyond SAP Blog Corner SAPADMIN Root Cause Analysis status update

Generated by Jive on 2013-02-21+01009

Now there is no remove button here but you could have already noticed the data is beingread from the SLD so you can remove the obsolete agents in the SLD

Picture 16

In order to remove obsolete agents from the SLD logon to the SLD using URLhttplthostgtltdomaingt5ltinstance nogt00sld with a user that sufficient rights to administratethe SLD

Next go to Administration (see picture 16)

Next click on Content Maintenance under header Content (see picture 17)

Tom Cenens Beyond SAP Blog Corner SAPADMIN Root Cause Analysis status update

Generated by Jive on 2013-02-21+010010

Picture 18

Now from the dropdown boxes choose All Classes for Subset and Diagnostics Agent forClass as shown in above screenshot (see picture 18)

Picture 19

This will provide you with a list of agents that exist in the SLD (see picture 19) Next to theabove shown columns you have hostname reference and more details (since it is sensitiveinformation I cannot show it) To remove an agent from the SLD mark the checkbox in frontof the agent and push the remove button which is available in the table toolbar

Improvements made

Overall there are some improvements compared to the previous version in terms ofinstallation and configuration you can directly register your Diagnostics Agent which takesaway the need to go through yet another registration process on SAP Solution ManagerInstead you can now directly start using the managed SAP system setup once your agentis installed SAPinst also checks the users and passwords entered during the installation forcorrectness which is also useful as in old versions you only noticed once the installation wasfinished

Tom Cenens Beyond SAP Blog Corner SAPADMIN Root Cause Analysis status update

Generated by Jive on 2013-02-21+010011

Concurrent mark sweep interpreter available

The interpreter which was missing for the Concurrent Mark Sweep garbage collection isalso ready and my customer message was answered by development to let me know thefollowing versions support the garbage collection output

Support of CMS GC output is now included in SolMan 701 with

- LMSERVICE SP07 Patch 3

- LMSERVICE SP08 Patch 1

and SolMan 71

- LMSERVICE SP01 Patch 1

Support list is an extract from a SAP customer message

End-note

I have to state that Irsquom getting the necessary support from by SAP so far helpsolve issues and that feedback is well received and actually being handled Thisshows in a number of notes documentation and patches

Itrsquos important to give feedback to help improve products Irsquom glad to see SAP iswilling to listen and work together to further enhance the product Of course theysaid themselves they want more open communication and a better connection tothe customer so this fits right into their recently announced strategy

You can expect to read more on this as improvements are made and pain points are beinghandled

88 Views

There are no comments on this post