IBM Operations Analytics - Log Analysis: User's Guide · 2 IBM Operations Analytics - Log Analysis:...

IBM Operations Analytics - Log AnalysisVersion 1.3.3

User's Guide

IBM

NoteBefore using this information and the product it supports, read the information in “Notices” on page 65.

Edition notice

This edition applies to IBM Operations Analytics - Log Analysis and to all subsequent releases and modificationsuntil otherwise indicated in new editions.

References in content to IBM products, software, programs, services or associated technologies do not imply thatthey will be available in all countries in which IBM operates. Content, including any plans contained in content,may change at any time at IBM's sole discretion, based on market opportunities or other factors, and is notintended to be a commitment to future content, including product or feature availability, in any way. Statementsregarding IBM's future direction or intent are subject to change or withdrawal without notice and represent goalsand objectives only. Please refer to the developerWorks terms of use for more information.

© Copyright IBM Corporation 2015.US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contractwith IBM Corp.

https://www.ibm.com/developerworks/community/terms/

Contents

About this publication . . . . . . . . 1Audience . . . . . . . . . . . . . . . 1Publications . . . . . . . . . . . . . . 1

Accessing terminology online . . . . . . . . 1Accessibility . . . . . . . . . . . . . . 1Tivoli technical training. . . . . . . . . . . 1Providing feedback . . . . . . . . . . . . 2Conventions used in this publication . . . . . . 2

Typeface conventions . . . . . . . . . . 2

Searching and visualizing data . . . . . 3Search UI overview . . . . . . . . . . . . 3Tabs . . . . . . . . . . . . . . . . . 4Side bar icons . . . . . . . . . . . . . . 4Search UI reference . . . . . . . . . . . . 5Changing the search time zone . . . . . . . . 8Searching data . . . . . . . . . . . . . . 9

Query syntax . . . . . . . . . . . . . 10Search results timeline . . . . . . . . . . 13List and Grid views . . . . . . . . . . 13Refining search results. . . . . . . . . . 14

Saving a search . . . . . . . . . . . . 16Saved searches . . . . . . . . . . . . . 16Visualizing data . . . . . . . . . . . . . 17

Creating charts and graphs . . . . . . . . 17Percentile statistical functions . . . . . . . 17Dashboards . . . . . . . . . . . . . 18

Search dashboards . . . . . . . . . . . . 21Alerts dashboard . . . . . . . . . . . 21

Custom Search Dashboards . . . . . . . . . 22Creating a Custom Search Dashboards . . . . . 23

Defining a Custom Search Dashboard . . . . 23Steps to create a Custom Search Dashboard . . 25Defining a search filter app . . . . . . . . 55Adding a shortcut to a Custom SearchDashboard to the Table view toolbar . . . . . 63

Notices . . . . . . . . . . . . . . 65Trademarks . . . . . . . . . . . . . . 67Terms and conditions for product documentation. . 67IBM Online Privacy Statement . . . . . . . . 6868Trademarks . . . . . . . . . . . . . . 68

© Copyright IBM Corp. 2015 iii

iv IBM Operations Analytics - Log Analysis: User's Guide

About this publication

This guide contains information about how to use IBM® Operations Analytics - LogAnalysis.

AudienceThis publication is for users of the IBM Operations Analytics - Log Analysisproduct.

PublicationsThis section provides information about the IBM Operations Analytics - LogAnalysis publications. It describes how to access and order publications.

Accessing terminology onlineThe IBM Terminology Web site consolidates the terminology from IBM productlibraries in one convenient location. You can access the Terminology Web site at thefollowing Web address:

http://www.ibm.com/software/globalization/terminology.

AccessibilityAccessibility features help users with a physical disability, such as restrictedmobility or limited vision, to use software products successfully. In this release, theIBM Operations Analytics - Log Analysis user interface does not meet allaccessibility requirements.

Accessibility features

This information center, and its related publications, are accessibility-enabled. Tomeet this requirement the user documentation in this information center isprovided in HTML and PDF format and descriptive text is provided for alldocumentation images.

Related accessibility information

You can view the publications for IBM Operations Analytics - Log Analysis inAdobe Portable Document Format (PDF) using the Adobe Reader.

IBM and accessibility

For more information about the commitment that IBM has to accessibility, see theIBM Human Ability and Accessibility Center. The IBM Human Ability andAccessibility Center is at the following web address: http://www.ibm.com/able(opens in a new browser window or tab)

Tivoli technical trainingFor Tivoli® technical training information, refer to the following IBM TivoliEducation Web site at http://www.ibm.com/software/tivoli/education.

© Copyright IBM Corp. 2015 1

http://www.ibm.com/software/globalization/terminology

http://www.ibm.com/able

http://www.ibm.com/software/tivoli/education

Providing feedbackWe appreciate your comments and ask you to submit your feedback to the IBMOperations Analytics - Log Analysis community.

Conventions used in this publicationThis publication uses several conventions for special terms and actions, operatingsystem-dependent commands and paths, and margin graphics.

Typeface conventionsThis publication uses the following typeface conventions:

Bold

v Lowercase commands and mixed case commands that are otherwisedifficult to distinguish from surrounding text

v Interface controls (check boxes, push buttons, radio buttons, spinbuttons, fields, folders, icons, list boxes, items inside list boxes,multicolumn lists, containers, menu choices, menu names, tabs, propertysheets), labels (such as Tip:, and Operating system considerations:)

v Keywords and parameters in text

Italic

v Citations (examples: titles of publications, diskettes, and CDsv Words defined in text (example: a nonswitched line is called a

point-to-point line)v Emphasis of words and letters (words as words example: "Use the word

that to introduce a restrictive clause."; letters as letters example: "TheLUN address must start with the letter L.")

v New terms in text (except in a definition list): a view is a frame in aworkspace that contains data.

v Variables and values you must provide: ... where myname represents....

Monospace

v Examples and code examplesv File names, programming keywords, and other elements that are difficult

to distinguish from surrounding textv Message text and prompts addressed to the userv Text that the user must typev Values for arguments or command options

2 IBM Operations Analytics - Log Analysis: User's Guide

Searching and visualizing data

This section outlines how to use the IBM Operations Analytics - Log AnalysisSearch workspace to search your indexed data and to display this data in chartsand dashboards.

To find the root cause of a problem experienced by users such as slowness or afailure, you can search through data such as log files, traces, configurationinformation, and utilization data. This type of search is iterative because the resultsfor one search might lead to a set of other searches. An example of iterative searchis finding the connection timeout in the error logs, which could lead you find theconnection pool utilization details.

Search UI overviewUse this topic to help you to get started with the Search UI.

The following screen shot shows the capabilities of the Search UI:

1. SidebarUse the UI icons on the side bar to open the Getting Started UI, a savedsearch, a search dashboard or the Administrative Settings UI.

2. Search Patterns paneThe Search Patterns pane lists the fields that are found in your search. Tofilter the search for a field, click on the field and click Search.

3. Discovered Patterns paneThe Discovered Patterns pane lists fields and values. To displaydiscovered patterns, you need to configure the source types in the datasource.

4. Timeline paneThe Timeline pane displays the current search results filtered by time. Todrill down to a specific time, click on a bar in the graph.

5. Timeline sliderUse the time line slider icon to narrow and broaden the time period.


6. Search boxEnter search queries in the Search field. When you click on a field in theSearch or Discovered Patterns pane, the query is displayed in this field.

7 Time filter listUse the time filter list to filter search results for a specified time range.

8. Data source filterUse the data source filter icon to filter search results for a specific datasource.

9. Table view / Grid viewUse the List View / Grid View icon to switch between both view. Use thelist view to identify search results quickly. Use the table view to displaysearch results in a tabular format.

TabsThe user interface consists of seven tabs. Read this topic to get an overview of thefunctions available under each one.

Getting StartedUse this tab to help you to get started with Log Analysis. You can viewdemonstrations and use sample data to help you to familiarize yourselfwith search and the other functions.

Data TypesUse this tab to create, edit, and delete collections, source types, rule sets,and file sets.

Data SourcesUse this tab to create, edit, and delete data sources and databaseconnections.

Hadoop IntegrationConfigure your Hadoop integration on the Configuration subtab andcheck the status of your current processes on the Integration Helathsubtab.

Roles Create, edit, and delete roles.

Users Create, edit, and delete users.

Server StatisticsDisplay the average and peak average data ingestion.

Side bar iconsUse the side bar to quickly navigate the user interface.

The following table explains the available icons.

Table 1. . Side bar icons

Icon Name Description

Getting Started icon Use guided demonstrationsand find links to usefulinformation.

Saved Searches icon Run saved and examplesearches.


Table 1. (continued). Side bar icons

Icon Name Description

Search Dashboards icon Run custom and samplesearch dashboards to viewcharts based on searchresults.

Administrative Settings icon Create and administrate thedata model, users, and roles.Display server statistics.

Manage Alerts icon AdvancedALLAltiVecB2BB2CBusinessC89C99CEPCloudscapeContent MgrDB2DeveloperDocumentumDomino.DocEnterpriseExpressFix Pack 1Fix Pack 2Fix Pack 3i5IDSiSeriesOracleOracle 10OrchestratorPanagonProfessionalpSeriesSQL ServerStartStudioSybaseSystem i5System p5System z9TelesalesWAS 5.0WAS 5.1WCS 5.1WC 5.4WorkgroupxSerieszSeriesStandard

Create andadministrate alerts. Thisfeature is only available inthe Standard Edition.

Search UI referenceUse this reference topic to help you to navigate the Search user interface (UI).

Buttons and fields on the Search UI

Table 2. Buttons and fields on the Search UI

Button or Field Name Description

Search button and field Search for a keyword orenter a search query.

Data Sources icon Filter the search for specificdata sources or groups ofdata sources.

Time filter icon Specify a relative or customtime filter.

Save Quick Search icon Save the search query andresults for later use. To viewthe saved searches, click theSaved Searches icon on theside bar.

Columns to be displayedicon

Filter the columns that aredisplayed in the Grid andTable views.

Plot chart icon Select the columns that youwant to graph and click thePlot Chart icon.

List View icon To open the List View, clickthis icon.

Grid View icon To open the Grid View, clickthis icon.

Searching and visualizing data 5

Time line graph

Table 3. Time line graph

Field, icon or button Description

Time line slider icon Filter the time that isdisplayed in the timelinegraph.

Y-axis Shows the number of logevents for each bar.

Bar Shows the number of logevents for the specific time.To view more details, click abar to drill down to moredetails about that time range.

Time zone button To change the time zone,click the time zone button.

Log Events Granularity Displays the granularity ofthe log records that aredisplayed. The level dependson the time that you havefiltered for. You can drilldown from years to monthsto days to hours to minutesand seconds.

Time Range Describes the time range thatis displayed in the timeline.

Plot Chart editor

Table 4. Buttons and fields on the Plot Chart editor


Generate Count check box To generate a count of theselected columns, ensure thatthis check box is selected.

Selected Columns A list of the columns thatyou selected in the Gridview.


Table 4. Buttons and fields on the Plot Chart editor (continued)


Plot Chart (Current PageData)

To create a chart of the dataon the current page, click thePlot Chart (Current PageData) button.

Plot Chart (All Data) To create a chart of all thedata in the results, click thePlot Chart (All Data) button.

Render Chart editor

Table 5. Render chart editor


Clear All To clear the graph and closethe window, click Clear All.

Create New Dashboard To create a new dashboardbased on the data in thegraph, click the Create NewDashboard icon.

Add Charts to ExistingDashboard

To add the chart data to adashboard, click the AddCharts to ExistingDashboard icon.

Hide Portlet icon To hide the graph, click theHide Portlet icon.

Settings icon To change the type of chartor the values that aredisplayed on the axes, clickthe Settings icon.

Close Portlet icon To close the graph and deletethe chart, click the ClosePortlet icon.

Chart Settings editor

Table 6. Chart settings editor


Render To create a chart, clickRender.

Visualization tab

Title Enter a name for the chart.

Chart Type Select the kind of chart thatyou want to use.

Parameters: x-axis Select the value that youwant to display on thex-axis.


Table 6. Chart settings editor (continued)


Parameters: y-axis Select the value that youwant to display on they-axis.

Query tab

Query String The query string that is usedby the search that generatedthe results.

Time Filters Select a time filter that forthe chart.

Datasource Filters Filter the chart data forspecific data sources.

Selected Columns The columns that the graphis based on.

Generate Count Indicates whether a countwas generated when thechart was plotted.

Change Time Zone dialog box

Table 7. Change Time Zone dialog box

Button or field Name Description

Time zone Select a time zone from thelist, or type an entry in thefield.

Default time zone selectioncheck box.

Select this check box to usethe specified time zone forall future searches.

Changing the search time zoneBy default, Log Analysis converts all times to the browser time zone.

The browser time may not match the time displayed in Log Analysis if there areissues with Daylight Savings Time.

For more information about this issue, see the Search time zone does not matchbrowser topic in the Troubleshooting guide.

If you do not want to use the default time zone, you can change it. To change thetime zone:1. Click the time zone.2. Select the city or region that represents the time zone that you want to use. For

example, if you are in Ireland and you want to set the time zone as GreenwichMean Time (GMT) you can select Europe/Dublin (Greenwich Mean Time.

3. If you want to use this time zone in subsequent searches, click the Use thistime zone as the default for all searches check box. This step is optional.


When you select a default time zone, you need to use the default time zonethat is most commonly used by Log Analysis users because this helps LogAnalysis to process log records more quickly and efficiently.

4. To save your changes, click OK

Searching dataYou can search ingested data such as log files for keywords. Search results aredisplayed in a timeline and a table format.

Before you begin

Before you can search, you must first define a data source and ensure that the logfile is configured for consumption or you can load the sample data.

Procedure1. From the Search workspace , click the New Search or Add Search tab to open

a new search table. Enter the search query.2. Optional: You can filter data source by name, description, host name, log path,

or tags or enter * to do a wildcard search. To limit the extent of the search toan individual data sources and any descendant data sources, select a leaf node

from the Data Sources tree ( ).

3. In the Time Filter pane, click the Time Filter list ( ) and selectthe time period for which you want to search. Select Custom to specify a starttime and date, and an end time and date for your search.

4. In the Search field, type the string for which you want to search in the log files.To view distribution information for all logs, in the Search field, type thewildcard character (*).To search for a partial string, type an asterisk (*) at the start and end of yoursearch string. For example, to search for strings that contain the phrasehostname, type *hostname*.To narrow your search based on a service topology, type the service topologycomponent on which you want to base your search, followed by a colon (:),followed by your search string. For example, service:DayTrader.

5. Click Search. The first search you perform after the IBM Operations Analytics -Log Analysis processes have been restarted might take longer to complete thansubsequent searches.The user interface refreshes every 10 seconds. The updated results aredisplayed in the progress bar.

Maximum search results: The search returns a maximum of 1000 records bydefault. This limit applies only to raw searches and not facet queries. This limitcan be configured in unitysetup.properties file property:MAX_SEARCH_RESULTS=1000. Do not to use a high value for theMAX_SEARCH_RESULTS parameter. When a large number of results are returned, itdegrades search performance.

Results

A graph displaying the distribution of matching events in the log is displayed. Logrecords containing a match for your search term are also displayed in Table view.


When you search for a specific term, the term is highlighted within the individuallog records to facilitate faster analysis. If you search for a partial term, each termthat contains the search phrase is highlighted. Fields that contain only taggedvalues, in other words values that are contained within angled brackets (<>), arenot highlighted. If a field contains values that are tagged and values that are nottagged, the tagged terms are removed and the remaining terms are highlighted asappropriate.

If your search spans data that is stored in the archive, IBM Operations Analytics -Log Analysis displays the initial results while it retrieves the rest of the data. Youcan interact with the initial search results while IBM Operations Analytics - LogAnalysis generates the search results. The progress bar displays the searchprogress.

To display the latest results during the search, click We have more results for you.To stop the search, close the tab. To start another search while you are waiting forthe first search to complete, click the Add Search tab.

What to do next

If you want to load data that contains tags and want to keep the tagging, you candisable highlighting. To disable highlighting:1. Open the unitysetup.properties file.2. Locate the ENABLE_KEYWORD_HIGHIGHTING property and set it to false.3. Save the file.4. To restart IBM Operations Analytics - Log Analysis run the following command

from the <HOME>/IBM/LogAnalysis/utilities directory:./unity.sh -restart

Related concepts:Loading and streaming dataBefore you can perform a search on log or other data, you must first load the datainto IBM Operations Analytics - Log Analysis. When the file is loaded the data isindexed and is then available to be searched.Data Source creationYou create data sources to ingest data from a specific source.“Query syntax”This section describes the combination of words, keywords, and symbols that youcan use when searching for phrases using IBM Operations Analytics - LogAnalysis.

Query syntaxThis section describes the combination of words, keywords, and symbols that youcan use when searching for phrases using IBM Operations Analytics - LogAnalysis.

The query syntax is based on the Indexing Engine query syntax. For moreinformation, see:

https://wiki.apache.org/solr/SolrQuerySyntax

Indexing Engines use a number of different query parser plug-ins. Log Analysissupports the Lucene query parser plug-in. For more information about the Lucenequery syntax, see:


https://wiki.apache.org/solr/SolrQuerySyntax

http://lucene.apache.org/core/5_1_0/queryparser/org/apache/lucene/queryparser/classic/package-summary.html

Standard keywords and operatorsThis topic lists the keywords and operators that you can use when searching inIBM Operations Analytics - Log Analysis.

Note: The operators such as AND and OR, which are part of this query syntax, arecase sensitive. You need to use capitals for these operators.

OR This is the default operator. Either term or expression must be matched inthe results. A variation to this keyword is or. For example, to search for aspecific severity or message classifier, enter severity:M ORmsgclassifier:"WLTC0032W".

+ To get AND like functions, use the + operator. You must add + as a prefixto these queries. For example, to search for a specific severity and messageclassifier, enter +severity:W +msgclassifier:"WLTC0032W".

AND As an alternative to the + operator, you can use the AND operator. Forexample, to search for a specific severity and message classifier, enterseverity:W AND msgclassifier:"WLTC0032W".

“” Enables you to group individual terms into phrases that are searched for asa unit. For example, “document clustering”.

() Enables you to group expressions to guarantee precedence. For example,document AND (cluster OR clustering).

* Wildcard operator that can be replaced in the returned value with anumber of characters. This can be either passed as an operator to thesources or expanded when the meta.wildcard-expand option is turned on.For example, test* might return test, tests or tester. You can also use thewildcard in the middle of the search term. For example, t*est.

Note: You cannot use this wildcard as the first character in a search. Forexample, you cannot use *test.

? Wild character operator that can be replaced in the returned value with asingle character. This can be either passed as an operator to the sources orexpanded when the meta.wildcard-expand option is turned on. Forexample, te?t might return text or test.

Note: You cannot use this wildcard as the first character in a search. Forexample, you cannot use ?test.

+ Must operator. Forces the use of a keyword. For example WAS +and DB2searches for strings that contain the keyword and.

field: Enables you to restrict your query to a specific field. For example, ID:123Aor msgclassifier:“WLTC0032W”. These operators are activated for everyfield defined in your syntax.

By default, the search engine supports the title field. When you arecreating a search collection, you can extract any number of contents, foreach document, and relate these contents to searchable fields. This isspecified in the form of the source associated with each collection.

NOT The specified term or expression must not be matched in the search results.Variations to this keyword are ! and -. For example, to search for logrecords that contain WAS ID but that do not contain DB2 ID, enter "WAS ID"NOT "DB2 ID".




Note: You cannot use this operator for a single term.

Additional keywords and operatorsThis topic lists additional keywords that are more specific to the search andindexing operations performed by the search engine.

Range searches

To search for records in a range, use a range query. Range queries caninclude the terms in the range or they can exclude them. To include thequery range terms, use brackets, for example:[<search term> TO <search term>]

To exclude the query range terms, use braces, for example:{<search term> TO <search term>}

Results are returned in lexicographical order.

For example, to search for all the log records modified on or between twodates, enter:mod_date:[20020101 TO 20030101]

The search returns all the log records that have been modified in 2003, thatis all the log records where the mod_date field is within the specified range.

You can also use range queries to search for fields that are not dates. Forexample, to search for all the log records that contain an ID between A toD but that do not include A or D, enter:title:{A TO D}

DateMath queries

To help you to implement more efficient filter queries for dates, you canuse DateMath queries.

For example, here are 4 possible DateMath queries:v timestamp:[* TO NOW]

v timestamp:[1976-03-06T23:59:59.999Z TO *]

v timestamp:[1995-12-31T23:59:59.999Z TO 2007-03-06T00:00:00Z]

v timestamp:[NOW-1YEAR/DAY TO NOW/DAY+1DAY]

For more information, see the DateMathParser topic in the Lucenedocumentation at:

http://lucene.apache.org/solr/5_1_0/solr-core/org/apache/solr/util/DateMathParser.html

Escaping special charactersRegular expression or regex queries are supported by the query syntax.

Log Analysis supports escaping for the following special characters:+ - && || ! ( ) { } [ ] ^ " ~ * ? : \ /

To escape a special character, use a back slash (\) before the special characters. Forexample, to search for the query (1+1):2, enter:$1\+1$\:2

To find multiple terms, use brackets. For example, to search for moat and boat,enter:




/[mb]oat/

Example query: Search for a keyword in a specified rangeThis example search query searches for a keyword in a specified time range. Youcan use queries like this one to search for a keyword in a specified range.

You want to search for all the instances of the error code 6543 in the SUMMARY fieldwith a response time less than 5 seconds. For example:fieldName:SUMMARY

fieldContents: "Transaction 12345 has failed with response time of 10seconds and error code of 6789."

You enter the following query. It specifies the summary field and a query for thetime range:"query" : "SUMMARY:/.*\\s([6-9]|\\d\\d+)\\ssecond.*6789\\./"

Search results timelineThe search results timeline displays a graph showing the distribution of log eventsover a time period.

You can use the timeline slider to view the logs for a specific duration. You canzoom in and out to change the range of the data displayed. If there are a largenumber of dates in the log file, the timeline might display them as ### rather thandisplaying the dates. Use the timeline scroller to zoom in and display theappropriate date information.

The Timeline does not display data at the seconds level for data ingested usingIBM Operations Analytics - Log Analysis Version 1.1. A message is displayedindicating that the data was indexed using a previous version of IBM OperationsAnalytics - Log Analysis. For this data, the Timeline cannot display information formultiple events that occur in time periods of less than one minute.

List and Grid viewsLog records are displayed in both in a grid view and a list view. The default viewis the List view. Log records are displayed in the grid view can be ordered bycolumn for easy analysis. This view can be customized and used to displayinformation in a range of ways:

Sorting in Grid viewYou can also sort the information in the table columns by clicking on thecolumn header. Not all columns are sortable. The Index configurationdetermines the fields that can be sorted.

The _datasource field is an internal field and cannot be sorted or used forcharting purposes. If you want to sort your data by data source or tocreate a chart, create a field in the Index configuration for this purpose.This field can be used for sorting and in charts.The order in which the fields are displayed is governed by the associatedindex configuration and source type. You can use the Index Configurationeditor to add new fields or adjust the order of existing fields.For more information about editing index configuration, see the Editing anindex configuration topic in the Administrating and Installing Guide.


Toggle viewsClick List View or Grid View button to toggle between views. In each ofthese views, the button displayed allows you to toggle to the alternativeview.

Customizing the columns displayedTo configure Grid view to display only the columns that you require, clickthe Select Columns icon on the Grid view toolbar, remove the columnsthat you do not want to display, and click OK.

Display a chart of your resultsYou can display the distribution of values for a number of the columns asa chart. To display the chart, select the column and click the Plot Columnicon on the Grid view toolbar. The distinct values used to plot the chartcan be viewed as hover help for each chart sector. The Plot feature isavailable for some values only. Where available, the Plot Column button isactive when the columns are selected.

If the chart title contains a loading icon, the chart is loading data from thearchive. The chart is automatically updated when all the searches arecomplete. If you log out before the search is completed, the search stops.

Running a Custom Search Dashboard from the Grid view toolbarIf you have configured a shortcut to a Custom Search Dashboard, click theicon on the toolbar to launch the Custom Search Dashboard.

Using a Custom Search Dashboard to display selected data from a column orcells If you have created the required Custom Search Dashboard, you can select

and display the contents of a column or individual cells. To display thedata, select a column or individual cell in Grid view and then launch theapplication. If you select a column, only data from the currently displayedpage is displayed by the application.

Related concepts:“Custom Search Dashboards” on page 22Custom Search Dashboards allow you to create and execute custom scripts anddisplay the output of those scripts in a dashboard.

Refining search resultsYou can refine the search results.

You can narrow your search, by adding extra criteria in the search field. Forexample, the string severity : E returns log lines that contain errors. Alternatively,you can perform a free text search for a value in a column. All of the log lines thatcontain that text are returned. If more than 100 log lines are returned, click thearrows to view more log lines.

Note: If a host file contains the character sequence ::1 next to the host name, ::1might be displayed as the value in the sourceip column.

You can also refine your search in these ways:

Search Patterns

To refine your search, use the values in the Search Patterns pane. For each newsearch, the list of fields with which you can filter your search is updated and listed


in the Search Patterns pane. The number of occurrences of each value that is foundis displayed with each unique keyword added as a child node. Click a keyword toadd it to the Search field.

The keyword is added in the format field:"value". You can add multiplekeywords to refine your search. If you want to run an OR query, type the word ORbetween each added keyword search string. When you add all of the searchcriteria, click Search to return log lines that contain the values that you specified.

Discovered Patterns

When you search a data source that has been configured with a Source Type thatuses the Generic annotator, the results of the search are listed in the DiscoveredPatterns pane.

For each new search, the list of fields with which you can filter your search isupdated and listed. The counts in the Discovered Patterns pane indicate thenumber of records that contain a specific key or key-value pair. A key-value pairmight occur multiple times in a record, but the total reflects the number of recordsin which the key-value pair occurs. The count of the value of nodes in a key-valuepair tree might exceed the key count when multiple values occur for the same keyin a single record.

Click a keyword to add it to the Search field. The keyword is added in the formatfield:"value". You can add multiple keywords to refine your search. If you wantto run an OR query, type the word OR between each added keyword search string.When you add all of the search criteria, click Search to return log lines that containthe values that you specified.

Data Source filtering

Refine your search by selecting a Data Sources leaf node. When you select a leafnode in the Data Sources tree, your search is refined to search only that datasource and any descendant data sources. The Data Sources tree is defined byselecting a service topology node when you configure your data source. For moreinformation, see Editing groups in the service topology JSON file.

Time Filters

Us the Time Filters list to refine your search based on a selected time period.Select a value from the list to limit the search period. The time period chosen limitsthe search time period based on the log entries. For example, choosing Last Hourlimits the search to the final 60 minutes of log file entries.

Selecting a timeline value

Click a value in timeline to refine your search based on that value. Log events canbe visualized up to second-level granularity.

Selecting a time zone

To change the time zone that is used in one or all of your searches, click theBrowser time link in the timeline chart. For more information about changing thetime zone, see Changing the search time zone in the Searching and visualizing dataguide.


Saving a searchAfter you search for a keyword or series of keywords, you can save your search sothat you can run it again at a later time. The searches you save are added to theQuick Searches pane.

About this task

Any directories that you created to organize your Quick Searches cannot bedeleted. The directory structure is maintained.

Procedure

To save a search:1. In the Search workspace, click the Save Quick Search icon. The Save Quick

Search dialog box is displayed.2. Enter a value in the Name and Tag fields. Adding a tag allows you to contain

similar searches within a folder.3. (Optional) Specify a time range as an absolute or relative time. The default

option is relative time.4. Click OK. The search is saved to the Save Quick Search pane.

What to do next

To use a saved search pattern, browse to the saved search in the Quick Searchespane and double-click the search pattern that you want to launch. You can alsoedit and delete the search from the right-click menu.

Saved searchesTo display a list of saved searches, click the Saved Searches icon.

The following saved searches are available by default after you install the sampledata:

sample WAS System OutExample search that displays results from WebSphere® Application Server.

sample DB2 db2diagExample search that displays results from DB2®.

sample MQ amqerrExample search that displays results from IBM MQ.

sample Oracle alertExample search based on alerts for the Oracle sample data.

sample App transaction logExample search based on the sample application's transaction log.

sample Omnibus eventsExample search based on the sample Omnibus events.

sample Windows OS eventsExample search based on the sample Windows OS events.


Visualizing dataYou can create charts and graphs that help users to process information quicklyand efficiently.

Creating charts and graphsAfter you select one or more columns in the grid view, you can use the PlotColumn button to create charts to display the results.

About this task

To create a chart that is based on a specific field, you can create a search query andplot a chart based on the results. If your query returns blank fields, the chart mightnot display. To ensure that this does not happen, use queries that return data forthe specific field. For example, to search for the severity field, enter severity:*TO*.

Procedure1. In the Search workspace, select Grid view.2. Select one or more columns.3. Click the Plot Column icon in the Grid view toolbar. The Plot Chart UI is

displayed.4. To display counts for the selected columns, select the Generate Counts check

box.5. If the columns that you selected contain dates or numeric values, you can use

the Granularity field to specify the granularity. You can use this setting only forcolumns that contain dates or numeric values and can be filtered.

6. If the columns that you selected contain numeric values, you can also applystatistical functions on the values.

v AdvancedALLAltiVecB2BB2CBusinessC89C99CEPCloudscapeContent MgrDB2DeveloperDocumentumDomino.DocEnterpriseExpressFix Pack 1Fix Pack 2Fix Pack 3i5IDSiSeriesOracleOracle 10OrchestratorPanagonProfessionalpSeriesSQL ServerStartStudioSybaseSystem i5System p5System z9TelesalesWAS 5.0WAS 5.1WCS 5.1WC 5.4WorkgroupxSerieszSeriesEntry If you are using the Entry Edition, you can use the sum, min,max, avg, and count functions.

v AdvancedALLAltiVecB2BB2CBusinessC89C99CEPCloudscapeContent MgrDB2DeveloperDocumentumDomino.DocEnterpriseExpressFix Pack 1Fix Pack 2Fix Pack 3i5IDSiSeriesOracleOracle 10OrchestratorPanagonProfessionalpSeriesSQL ServerStartStudioSybaseSystem i5System p5System z9TelesalesWAS 5.0WAS 5.1WCS 5.1WC 5.4WorkgroupxSerieszSeriesStandard If you use the Standard Edition, you can use the missing,sumOfSquares, stddev, and percentile functions.

7. To plot the chart on 100 or less of the records that you selected, click Plot Chart(Current Page Data).

8. To plot the chart on all the indexed data, click Plot Chart (All Data). If one ormore of the fields in the selected columns is not filterable, the charts are onlyplotted if the total number of records is less than 1000. To change this setting,you must modify the MAX_DATA_FACETS_IN_CHART property in theunitysetup.properties file.

Results

The graph is rendered. To change the graph type, use the Edit icon.

If the chart title contains a loading icon, the chart is loading data from the archive.The chart is automatically updated when all the searches are complete. If you logout before the search is completed, the search stops.

Percentile statistical functionsYou can use the statistical function to return values for specified percentiles.


You can use the Search REST API or the user interface to create percentilestatistical functions.

For example, to query the maximum and minimum results for the 50th, 95th, and99th percentiles with the Search REST API, you enter "stats": ["min", "max","percentile,50,95,99"]. The facet response is:

{"min": 10,"max", 1000,"percentile":{

"50": 100,"95": 200,"99": 225

}}

Percentile queries are not calculated incrementally, unlike the other queries that areused in Log Analysis. This fact means that the query needs to run over the entiretime range before it can return any results.Log Analysis limits the number ofasynchronous windows that can run simultaneously for this function. Thisproperty is set in the MAX_NON_INCREMENTAL_WINDOWS property intheunitysetup.properties. The default value is 2.

For example, if you specify a percentile query based on a time range from August1 2015 to August 10 2015 and MAX_NON_INCREMENTAL_WINDOWS=5 andCOLLECTION_ASYNC_WINDOW=1d, only the most recent 5 days of data that is returnedby the query are considered for percentile evaluation.

DashboardsYou can use dashboards to collate multiple charts, which are created duringproblem diagnosis, on a single user interface (UI).

For example, imagine an organization uses IBM Operations Analytics - LogAnalysis to monitor all the server logs that it generates. The system administratorwants to be able to view the most critical errors, the highest severity errors, andthe total number of errors on a single UI. To facilitate this scenario, you create adashboard that is called System Admin and you add the charts that show therequired information to it.

The data that is displayed on the dashboards is based on charts. For moreinformation, see the Charts topic under Custom Search Dashboard > Steps tocreate a Custom Search Dashboard > Application files in the Extending IBMOperations Analytics - Log Analysis section.

Sample dashboards

Sample dashboards are included as part of the sample content for the followingCustom Search Dashboard samples:v Sample_EventInsightpack_v1.0.0.0v Sample_AppTransInsightpack_v1.0.0.0v Sample_weblogInsightpack_v1.0.0.0v WASInsightPack_v1.1.0.3

Creating dashboardsYou can create a dashboard to visualize data from multiple sources on a single UI.


About this task

This procedure describes how to create a dashboard and chart. You can also add achart to an existing dashboard. To add a chart to an existing dashboard, click AddCharts to an Existing Dashboard. Select a dashboard from the list.

Use the drill-down feature to search the data for records that correspond to thearea of the chart that you select. When you drill down on a field in a chart, a newsearch is created that is based on the field that you selected.

The drill-down feature is only supported for dashboards that are created in the UI.The drill-down feature is not supported in some cases where the underlying chartquery does not support it. For example, if the query searches two date fields.

Procedure1. Open an existing search UI or click Add New Search to create a new search UI.2. Switch to the Grid View and plot the charts that you would like to include in

the dashboard. You cannot use more than eight charts in a single dashboard.For more information, see “Creating charts and graphs” on page 17.

3. To plot the charts that you would like to include in the dashboard, select thecolumns that you are interested in and click the Plot column icon. Click PlotChart (All Data).If you want to use the drill-down feature, you must click Plot Chart (All Data).You cannot use the Plot Chart (Current Page Data) button. The drill-downfunction is not supported for this option.You can also run a number of statistical operations on the selected columns.Select one of the following functions from the Summary Function drop-downlist.

min The minimum values in a field.

max The maximum values in a field.

sum The sum of the values in a field.

avg The average of the values in a field.

count The count of the values in a field.

missingThe number of records for which the value for a field is missing.

sumOfSquaresSum of the squares of the values in a field.

stddevThe standard deviation of the values in a field.

4. To create the dashboard, click the Create New Dashboard button. Enter a nameand a tag. The tags are used to define groupings.

5. Save the dashboard.

What to do next

After the dashboard is created, a Custom Search Dashboard is automaticallygenerated that represents the dashboard. You can view the Custom SearchDashboard and dashboard on the Search UI under Search Dashboard >Dashboards.


To view the visualization and query settings for the charts that you added to acustom dashboard, click Search Dashboard > Dashboards > Dashboard name.Select the required chart. Click the Settings icon. There are two tabs, the Queryand the Visualization tabs.

Information about the query that provides the information that is visualized in thechart is displayed on the Query tab. This query is saved when the chart is created.To change the time filter from the default relative setting to match the absolutetime that is used by the Search UI, open the Query tab and select the setting fromthe list. To view the effect of changing this setting without changing thedashboard, click Render.

The chart type and parameters are detailed on the Visualization tab. These settingsare the settings that you made when you created the chart. For more informationabout these settings, see the Application files and Charts topics in the Custom SearchDashboards section of the Extending IBM Operations Analytics - Log Analysis guide.

To ensure that your dashboard displays current information, you can use theauto-refresh feature to regularly refresh a dashboard at scheduled intervals. Formore information on configuring automatic dashboard refreshes, see theConfiguring automatic refreshes for new dashboards topic in the Administering IBMOperations Analytics - Log Analysis guide.

Deleting search dashboardsIf you no longer need a dashboard, you can delete it.

About this task

There are two types of dashboard that is displayed in the Search Dashboards liston the UI, dynamic dashboards and Custom Search Dashboards. Dynamicdashboards do not contain any custom logic and the type value that is defined inthe associated JSON file is DynamicDashboard. Custom Search Dashboards arecustomized dashboards that are installed as part of an Insight Pack. They docontain custom logic.

The delete functions differently for each type. When you delete a dynamicdashboard, the dashboard and all related data are deleted. When you delete aCustom Search Dashboard, the Custom Search Dashboard file extension is changedto .DELETED for deletion by the IBM Operations Analytics - Log Analysisadministrator.

Procedure1. Open the Search UI.2. Open the Search Dashboards list and select the dashboard that you want to

delete.3. Right-click the dashboard and click Delete. Confirm that you want to delete it

when prompted.

Results

If the dashboard is a dynamic dashboard, the dashboard and associated data isdeleted. If the dashboard is a Custom Search Dashboard, the Custom SearchDashboard file extension is changed to .DELETED. You can contact the IBMOperations Analytics - Log Analysis administrator and ask them to delete theCustom Search Dashboard if appropriate.


Search dashboardsTo display a list of search dashboards, click the Search Dashboards icon on theside bar.

The Dashboards group contains the following search dashboards:

sample-events-hotspotsDisplays example hot spot reports for the sample events.

WAS Errors and Warnings DashboardDisplays example reports for errors and warnings that are generated by thesampleWebSphere Application Server application.

Sample-Web-AppDisplays example reports for errors and warnings that are generated by thesample web application.

The DB2AppInsightPack group contains the following search dashboards:

DB2 Information LinksDisplays useful information links for more information about DB2.

DB2 TroubleshootingDisplays example reports for DB2.

The ExpertAdvice group contains the following search dashboard:

IBMSupportPortal-ExpertAdviceDisplays search results based on your searches.

The WASAppInsightPack group contains the following search dashboards:

WAS Information LinksDisplays useful information links for more information about WebSphereApplication Server.

WAS Errors and WarningsDisplays example reports based on errors and warnings in WebSphereApplication Server.

The WindowsOSEventsInsightPack group contains the following searchdashboard:

Windows Events Log DashboardDisplays example reports that are based in event data from the Windowsoperating system.

Fix Pacck 1

The alertsInsightPack group contains the following search dashboard:

Alerts DashboardDisplays alert data based on the alerts that are created on the ManagerAlerts UI. The charts display the alert information grouped by type anddata source and over time.

Alerts dashboardFix Pacck 1

Use the Alerts dashboard to view information based on the alerts that you createin Log Analysis.


This feature is not available in the Entry Edition.

The data that is displayed on the dashboard is based on the last time data fromone of the data sources used for alerts was loaded into Log Analysis. This meansthat there is a delay between changes to your alerts and the information that isdisplayed on the dashboard. In this case, you need to wait until data is loaded intoLog Analysis again to see the latest data.

If the dashboard does not display any data, ensure that the time that you specifiedmatches a period when alerts were created.

Prerequisites

You must install IBM Operations Analytics - Log AnalysisFix Pack 1.

You must create alerts in Log Analysis. For more information, see .

Using the dashboard

The dashboard displays information on four graphs:

TOTAL ALERTS BY TYPEDisplays the total number of alerts grouped by type.

TOTAL ALERTS BY TYPE OVER TIMEDisplays the total number of alerts grouped by type for the chosen timerange.

TOTAL ALERTS BY DATASOURCEDisplays the total number of alerts grouped by data source.

TOTAL ALERTS BY DATASOURCE OVER TIMEDisplays the total number of alerts grouped by data source for the chosentime range.

Custom Search DashboardsCustom Search Dashboards allow you to create and execute custom scripts anddisplay the output of those scripts in a dashboard.

To generate a Custom Search Dashboard, you create a JSON application file in the<HOME>/IBM/LogAnalysis/AppFramework/Apps directory. You can create subdirectories in this directory to organize your applications.

After you have created your Custom Search Dashboard, the Custom SearchDashboards pane, in the Search workspace, displays the list of Custom SearchDashboards in the folder structure that you have specified.

To run a Custom Search Dashboard, locate it in the Custom Search Dashboards,right-click and click Execute.

For information about developing and customizing Custom Search Dashboards, seethe Extending section of the documentation on the IBM Knowledge Center.

Expert advice

Expert advice is a Custom Search Dashboard that provides links to contextuallyrelevant information to allow you to quickly resolve problems. Using the Expert


advice Custom Search Dashboard, you can select any column or cells in Grid viewand launch a search of the IBM support portal (IBMSupportPortal-ExpertAdvice.app). The Custom Search Dashboard searches for matches to uniqueterms contained in the column that you have selected. This Custom SearchDashboard can be launched from the Custom Search Dashboards panel in the leftnavigation pane of the Search workspace.

To increases the likelihood of success, the Custom Search Dashboard removes datathat is specific to your environment for each search term. For example, a logmessage that contained the search string unable to access jarfile/myMachine/foo/bar/foobar.jar is not likely to return a specific match as theserver path is likely to be specific to a user. This is abbreviated to unable toaccess jarfile to ensure better search results. The criteria used to exclude datacan be configured. For more information about configuring the Expert adviceCustom Search Dashboard, see the Configuring IBM Operations Analytics - LogAnalysis section of the IBM Knowledge Center.

To launch the Expert advice Custom Search Dashboard for the data returned by asearch, select a column or cell of data in Grid view, right-click and click Execute tolaunch the IBMSupportPortal-ExpertAdvice.app Custom Search Dashboard.

Creating a Custom Search DashboardsYou can use Custom Search Dashboards to extract data from IBM OperationsAnalytics - Log Analysis or from any external source and present that data in auseful format such as a chart.

A Custom Search Dashboard must include a script to generate data, anyparameters required by the script, and a specification to define how that data isdisplayed. There are two types of output that are generated by Custom SearchDashboards:

Dashboard dataCreate a IBM Operations Analytics - Log Analysis dashboard to displaycharts based on the data generated by a Custom Search Dashboard. Youcan also use a dashboard to display HTML content.

Search filtersGenerate keywords and populate Configured Patterns widget to drill downthrough a search.

When using the Insight Pack tooling in Eclipse, there is a folder calledsrc-files/unity_apps in the Insight Pack project, specifically for Custom SearchDashboard files. There are sub folders for apps, chartspecs, and templates tocontain application definitions, chart specifications, and template definitions,respectively.

Defining a Custom Search DashboardTo create a Custom Search Dashboard you must create an application file and alsocreate a script that extracts data to be displayed in your Custom Search Dashboard.Where appropriate, you can also create a template file that contains an array ofJSON objects. Templates define the structure for Custom Search Dashboards andalso allow you to share common components across Custom Search Dashboards.


Prerequisite knowledge

Creating a Custom Search Dashboard requires that you have a knowledge of theseareas:v Coding in one of the accepted formats: Python, Shell scripting, or Java

programmingv JSON development

Custom Search Dashboard components

These are the components that are required for a Custom Search Dashboard:

ApplicationThe application file is a JSON file that contains the configuration for yourCustom Search Dashboard. The JSON application file is saved with a .appfile extension and must be located in the <HOME>/AppFramework/Apps/directory or in a directory within this directory. Any directory structureyou create is reflected in the Custom Search Dashboards pane in the Searchworkspace.

The application file contains:1. References to the script that defines your Custom Search Dashboard2. Parameters that must be passed to the script to generate data, such as

user credentials and host name.3. Specifications that define the layout of your Custom Search Dashboard

and the charts that are displayed, including the chart type, and/or theHTML to be displayed.

Script The script that you run to define your Custom Search Dashboard can be ashell script, a python script or a Java program. If you are running a Javaprogram, you must bundle it as an executable JAR file.

The script performs the actions:1. Generates an HTTP POST request to a IBM Operations Analytics - Log

Analysis URL.2. Extracts the JSON that contains the data generated by the search in the

HTTP POST request.3. Generates JSON string representing HTML and/or data that the Charts

can process.

Scripts can also be written to extract data from an external source insteadof indexed data.The script must be saved to the same folder as the application file and iscalled by the application file.

Note: For IBM Operations Analytics - Log Analysis 1.2.0.3 and later, theIBM Operations Analytics - Log Analysis query syntax is based on theIndexing Engine query syntax rather than the IBM Data Explorer querysyntax, which was used in releases before 1.2.0.3. If your Custom SearchDashboard uses query syntax from a version of IBM Operations Analytics -Log Analysis that is older than 1.2.0.3, you must update the query syntaxused in your script to match the Indexing Engine based query syntax. Formore information, see Appendix A: Query syntax in the Searching Guide.

TemplatesTemplates contain JSON objects that are similar to the JSON used for the


application file. The template JSON objects contain an additional keynamed template with the value that is set to true. Each template canreference one or more custom scripts.

The templates are saved with a file extension of .tmpl to the<HOME>/AppFramework/Templates/ directory or a directory within thisdirectory. You must store the script file(s) referenced by the template in thesame folder as the template.

If an application file specifies a template, the script located in the samefolder as the template is executed. Templates can be used to create acommon structure for a group of applications and also to share commonconfiguration elements across applications.

Script parameters can be marked as mandatory by adding the parameterrequired and setting the value to true. You can also set a default value forany parameter.

Steps to create a Custom Search DashboardComplete these steps to create a Custom Search Dashboard.

Procedure1. The source log file you want to use must be loaded and processed by IBM

Operations Analytics - Log Analysis.2. Using python, shell scripting, or Java, create a script. The script and its

corresponding application or template file must reside in the same directory:<HOME>/AppFramework/Apps. If no value is specified for the type parameter atthe top of the application file, the application file runs the script from the samefolder as the application file.

Insight Packs and applications: If you want to include the application in anInsight Pack project, the script and the associated application file must be in thesame folder within the project: src-files/unity_apps/apps under the projectfolder.In the script, you need to specify the actions:a. Connect to IBM Operations Analytics - Log Analysis .b. Use an HTTP POST to run a search request to a IBM Operations Analytics

URL for JSON data. The query uses the same method as the queries youcan run in the Search Workspace. For the query, use the JSON format forSearch requests as described in the Search runtime API. If you need tonarrow down the returned data further, you can parse the data within thescript.

c. Format the returned JSON data into a structure that can be consumed bythe charts.

3. Create a JSON application file and save it to the <HOME>/AppFramework/Apps/directory or a sub-directory within this directory.If you want to include the application in an Insight Pack project, create theJSON application file in the src-files/unity_apps/apps folder. The applicationfile references your script and specifies the chart type and parameters for thedashboard display.

Note: If you are using the Insight Pack Tooling, build the Insight Packcontaining the application, and use the pkg_mgmt utility to install the InsightPack on your IBM Operations Analytics system.


4. From the Custom Apps pane in the Search workspace, run your applicationand determine if it is displaying your data as you intended. Amend the scriptand application file as required.

Refresh Custom Apps: If you do not see you application listed, refresh theCustom Apps pane in the Search workspace in order to list the newly installedCustom app.

Note: If you close a chart portlet, you must run the Custom Search Dashboardagain to reopen the chart.

5. (Optional) If you want to create a template for your application, create atemplate in the directory: <HOME>/AppFramework/Templates. If a template namehas been specified in the type field, the application file references that templateand executes the script in the same directory as that template. That is, theapplication file and script must be located in the same directory as the templatefile.

Insight Packs and templates: If you want to include the application in anInsight Pack project, the script, the application, and template file must reside inthe same in the folder within the project: src-files/unity_apps/templatesunder the project folder.

Related concepts:Search REST API overviewThe Search REST interface can be used to execute search queries. Search can beexecuted through a HTTP POST request on https:<host>:<port>;/Unity /Search.Related reference:Facet requestsDifferent types of facet requests are supported by USR, along with the JSON formatused to specify each type of facet request.JSON Format for Search RequestThe input JSON for a search request is a single JSON object.

ScriptsThe script that is executed can be written as a Python script, shell script, or a Javaapplication packaged as an executable JAR file. When you execute the application,the parameters are written to a temporary file in JSON format, and the file name ispassed to the script. The script reads the parameters from the file. And the scriptgenerates the output in the standard JSON format required for the dashboardspecification.

Within an Insight Pack project, scripts must reside in the same folder as theapplication or template that references it, either in src-files/unity_apps/apps orsrc-files/unity_apps/templates.

In the script, use an HTTP POST request to query a IBM Operations Analytics URLfor JSON data. The query uses the same method as the queries you can run in theSearch Workspace. For the query, use the JSON format for Search requests asdescribed in Search REST API. If you need to narrow down the returned datafurther, you can parse the data within the script.

Setting environment and connection in Python scripts

If you are using Python, you can import convenience methods in unity.py forinteracting with the IBM Operations Analytics server. unity.py is shipped withIBM Operations Analytics and can be found in PYTHONPATH . It contains methods


for connecting to the Operations Analytics servers, as well as making HTTPrequests that include the required Cross Site Request Forgery (CSRF) tokens.

To use unity.py, simply include it in an import statement. For example:from unity import UnityConnection, get_response_contenttry:

import jsonexcept ImportError:

import simplejson as jsonimport sys

To connect to the Operations Analytics server, create an instance ofUnityConnection() and call the UnityConnection.login() method. TheUnityConnection() object takes the parameters:v url - base URL to the Operations Analytics server; if you changed the default

port number during install, you should also change it herev username - username used for authenticationv password - password used for authentication

For example:# Create a new UnityConnection to the server and loginunity_connection = UnityConnection(’https://Myhost:9987/Unity/’,’unityadmin’, ’password’)unity_connection.login()

The parameters to the Python script are written to a file in JSON format, and thefilename is passed to the script. For example, the parameters can be passed in a filecontaining the following:{

"parameters": [{

"name": "search","type": "SearchQuery","value": {"filter": {"range": {

"timestamp":{"from":"01/01/2013 00:00:00.000 EST","to":"01/01/2014 00:00:00.000 EST","dateFormat":"MM/dd/yyyy HH:mm:ss.SSS Z"

}}

},"logsources": [

{"type": "logSource","name": "SystemOut"

}],

}},

]}

Here is an example of how the parameters can be retrieved and parsed:# Get the parametersif len(sys.argv) > 1:

filename = str(sys.argv[1])fk = open(filename,"r")data = json.load(fk)


parameters = data[’parameters’]

for i in parameters:if i[’name’] == ’search’:

search = i[’value’]for key in search.keys():

if key == ’filter’:filter = search[’filter’]

elif key == ’logsources’:logsource = search[’logsources’]

To make an http request that uses the Search runtime API, use theconnection.post() and get_response_content() methods. For example:request = {"logsources": logsource, // this is the value from the parameters"query": "*","filter": filter // this is the value from the parameters

}

response = connection.post(’/Search’, json.dumps(request),content_type=’application/json; charset=UTF-8’);content = get_response_content(response)

# parse the results in found in content...

Lastly, close the connection at the end of the Python script:# close the connectionconnection.logout()

Related reference:JSON Format for Search RequestThe input JSON for a search request is a single JSON object.Facet requestsDifferent types of facet requests are supported by USR, along with the JSON formatused to specify each type of facet request.

Example search request:

The search request retrieves JSON data and is the body in the HTTP POST request.

This is an example of a JSON search string that is the body HTTP POST request.{

"start":0,"results":1,"filter":{

"range":{"timestamp":{

"from":"5/31/2012 5:37:26.682 -0400","to":"5/31/2013 5:37:26.682 -0400","dateFormat":"MM/dd/yyyy HH:mm:ss.SSS Z"

}}

},"query":"severity:(W OR E)","sortKey":[

"-timestamp"],"getAttributes":[

"timestamp","severity"


],"facets":{

"dateFacet":{"date_histogram":{

"field":"timestamp","interval":"hour","outputDateFormat":"MM-dd HH:mm","nested_facet":{

"severityFacet":{"terms":{

"field":"severity","size":10

}}

}}

}},"logsources":[

{"type":"logSource","name":"/SystemOut"

}]

}

The request specifies:v The number of records returned, in this case it is "results":1. It is helpful to set

the number records to a low number when you are building and testing yourscript.

v The date/time range is from May 31 2012 to May 31, 2013:{"timestamp":{"from":"5/31/2012 5:37:26.682 -0400","to":"5/31/2013 5:37:26.682-0400","dateFormat":"MM/dd/yyyy HH:mm:ss.SSS Z"}}}

v The query searches for Warnings or Errors and the results are sorted bytimestamp. Only the timestamp and severity attributes are returned in theresults."query":"severity:(W OR E)","sortKey":["-timestamp"],"getAttributes":["timestamp","severity"],

v The logsource is set as the WebSphere Application Server SystemOut log."logsources":[{"type":"logSource","name":"/SystemOut"}]

v (Optional) Facet requests that you can use to create sums, time interval, or otherstatistical data for a given field/value pair. In this example there is adate_histogram facet with a nested terms facet. Within each time intervalreturned by the date_histogram facet, the term facet, called severityFacet,counts the number of each type of severity."facets":{"dateFacet":{"date_histogram":{"field":"timestamp","interval":"hour","outputDateFormat":"MM-dd HH:mm","nested_facet":{"severityFacet":{"terms":{"field":"severity","size":10}}}}}},

1,000 or more results and Custom Search Dashboard: When a query in a CustomSearch Dashboard returns more than 1000 records, you get only 1000 results back.The search result returned includes a field totalResults which shows total numberof matching results. Another field numResults gives the number of recordsreturned. You can check these values in the Custom Search Dashboard script andhandle the results accordingly.


Example JSON output from search request:

Run your script and review the JSON output. Adjust the script to get the outputyou need.

Search request output

When you post the HTTP Search request, it returns a JSON string in raw format. Itis helpful to open the raw output in a JSON editor. These samples show the outputfrom the example search in raw format and in the tabbed format displayed inJSON editors.

This sample is segment of the returned JSON in raw format.{"searchRequest":{"start":0,"results":1,"filter":{"and":[{"range":{"timestamp":{"from":"5\/31\/2012 5:37:26.682 -0400","to":"5\/31\/2013 5:37:26.682 -0400","dateFormat":"MM\/dd\/yyyy HH:mm:ss.SSS Z"}}},{"or":[{"phrase":{"logsource":"SystemOut"}}]},{"range":{"_writetime":{"dateFormat":"yyyy-MM-dd’T’HH:mm:ss.SSSZ","from":"2013-05-25T00:00:00.000-0400","to":"2013-05-31T05:48:18.415-0400"}}}]},"query":"severity:(W OR E)","sortKey":["-timestamp"],"getAttributes":["timestamp","severity"],"facets":{"dateFacet":{"date_histogram":{"field":"timestamp","interval":"hour","outputDateFormat":"MM-dd HH:mm","nested_facet":{"severityFacet":{"terms":{"field":"severity","size":10}}},"__usr_fast_date_query":"UTC"}}},"logsources":[{"type":"logSource","name":"\/SystemOut"}],"collections":["WASSystemOut-Collection1"]},"totalResults":805,"numResults":1,"executionInfo":{"processingTime":33,"searchTime":54},"searchResults":[{"resultIndex":0,"attributes":{"msgclassifier":"TCPC0003E","threadID":"00000000","message":"TCP Channel TCP_2 initialization failed. The socket bind failed for hostSystemOut.logSystemOut.logOneLiners SystemOut.logOneLinersNoTS unity_populate_was_log.shunity_search_pattern_insert.shwas_search_pattern.txt x and port 9080. The port may already be in use._LOG_2_","_writetime":"05\/27\/13 12:39:03:254 +0000","logsourceHostname":"fit-vm12-230","logRecord":"[10\/21\/12 19:57:02:313 GMT+05:30] 00000000TCPPort E TCPC0003E:TCP Channel TCP_2 initialization failed. The socket bind failedfor host SystemOut.log SystemOut.logOneLiners SystemOut.logOneLinersNoTSunity_populate_was_log.sh unity_search_pattern_insert.sh was_search_pattern.txt xand port 9080.The port may already be in use._LOG_2_","timestamp":"10\/21\/12 14:27:02:313 +0000","severity":"E","logsource":"SystemOut"}}],"facetResults":{"dateFacet":[{"label":"13-295-2012 UTC","low":"10-21 13:00","high":"10-21 14:00","count":425,"nested_facet":{"severityFacet":{"counts":[{"term":"W","count":221},{"term":"E","count":204}],"total":2}}},{"label":"14-295-2012 UTC","low":"10-21 14:00","high":"10-21 15:00","count":380,"nested_facet":{"severityFacet":{"counts":[{"term":"E","count":197},{"term":"W","count":183}],"total":2}}}]},"metadata":{"msgclassifier":{"filterable":true,"dataType":"TEXT","sortable":false},...""exceptionMethodName":{"filterable":false,"dataType":"TEXT","sortable":false},"severity":{"filterable":true,"dataType":"TEXT"

This sample shows the same results formatted in a JSON editor.{

"searchRequest": {"start": 0,"results": 1,"filter": {

"and": [


{"range": {

"timestamp": {"from": "5/31/2012 5:37:26.682 -0400","to": "5/31/2013 5:37:26.682 -0400","dateFormat": "MM/dd/yyyy HH:mm:ss.SSS Z"

}}

},{

"or": [{

"phrase": {"logsource": "SystemOut"

}}

]},{

"range": {"_writetime": {

"dateFormat": "yyyy-MM-dd’T’HH:mm:ss.SSSZ","from": "2013-05-25T00:00:00.000-0400","to": "2013-05-31T05:48:18.415-0400"

}}

}]

},"query": "severity:(W OR E)","sortKey": [

"-timestamp"],"getAttributes": [

"timestamp","severity"

],"facets": {

"dateFacet": {"date_histogram": {

"field": "timestamp","interval": "hour","outputDateFormat": "MM-dd HH:mm","nested_facet": {

"severityFacet": {"terms": {

"field": "severity","size": 10

}}

},"__usr_fast_date_query": "UTC"

}}

},"logsources": [

{"type": "logSource","name": "/SystemOut"

}],"collections": [

"WASSystemOut-Collection1"]

},"totalResults": 805,"numResults": 1,


"executionInfo": {"processingTime": 33,"searchTime": 54

},"searchResults": [

{"resultIndex": 0,"attributes": {

"msgclassifier": "TCPC0003E","threadID": "00000000","message": "TCP Channel TCP_2 initialization failed.

The socket bind failed for host SystemOut.log SystemOut.logOneLinersSystemOut.logOneLinersNoTSunity_populate_was_log.sh unity_search_pattern_insert.shwas_search_pattern.txt x and port 9080.The port may already be in use._LOG_2_",

"_writetime": "05/27/13 12:39:03:254 +0000","logsourceHostname": "fit-vm12-230","logRecord": "[10/21/12 19:57:02:313 GMT+05:30]

00000000 TCPPort E TCPC0003E:TCP Channel TCP_2 initialization failed.The socket bind failed for host SystemOut.logSystemOut.logOneLiners SystemOut.logOneLinersNoTS unity_populate_was_log.shunity_search_pattern_insert.sh was_search_pattern.txt x and port 9080.The port may already be in use._LOG_2_",

"timestamp": "10/21/12 14:27:02:313 +0000","severity": "E","logsource": "SystemOut"

}}

],"facetResults": {

"dateFacet": [{

"label": "13-295-2012 UTC","low": "10-21 13:00","high": "10-21 14:00","count": 425,"nested_facet": {

"severityFacet": {"counts": [

{"term": "W","count": 221

},{

"term": "E","count": 204

}],"total": 2

}}

},...

},"logsource": {

"filterable": true,"dataType": "TEXT","sortable": true

}}

}


Note: The TotalResults value shows the total number of matching results. ThenumResults value shows the number of returned results. The search requestspecified the return of one result. ("results":1).

JSON output from a script:

You can specify a Custom Search Dashboard with multiple charts with each chartpointing to a different data element. Data elements can be in the form of data orHTML. The script returns this JSON structure that contains the data that is used topopulate the charts, or HTML to be displayed.

You can also configure Custom Search Dashboards to return error messages in thefollowing format:{

"error_msg": "<error_message_string>"}

where <error_message_string> is the message that you want to display.

The following example shows the output parameters for a Custom SearchDashboard. The first data set contains the correct data that is used to render thechart. The second data set contains an error message.{

"data": [{"rows": [{

"userId": "116","count": 9

}],"fields": [{

"label": "userId","type": "TEXT","id": "userId"

},{

"label": "count","type": "LONG","id": "count"

}],"id": "DynamicDashboardSearch_0_1386061434000"

},{

"error_msg": "CTGLA2107E : Custom applications need to be configured forevery use case with details of data sources. No valid data sources were specifiedin the search request. Verify that the chosen data sources exist and that anyspecified tags contain data source descendants. For further information on how toconfigure the custom applications, refer to the documentation.",

"id": "DynamicDashboardSearch_1_1386061434000"}]

}

Example Script:

The full script example shown here contains all the required elements.

In this example script, the value for some elements are defined as variables insidethe script. For example, the elements of the search request such as the logsourceand query are defined as variables. Custom scripts should return date field dataonly in the supported formats for charts to render correctly. Supported formatsinclude yyyy-MM-ddTHH:mm:ssZ and MM/dd/yy HH:mm:ss:SSS Z.


from unity import UnityConnection, get_response_contentfrom urllib import urlencodeimport CommonAppMod

try:import json

except ImportError:import simplejson as json

import sysimport refrom datetime import datetime, date, time, timedeltaimport UnityAppMod

# Create a new Unity connection to the server and loginunity_connection = UnityConnection(’https://myhost:9987/Unity/’, ’unityadmin’,’unityadmin’)unity_connection.login()

########################################################### Calculates the current time, relative time, and forms timestamps##########################################################currentTimestamp = datetime.now()

# Currently chosen relative timestampnewTimestamp = currentTimestamp - UnityAppMod.getLastYear()

# Format the timestampcurrentFormatted = UnityAppMod.formatTimestamp(currentTimestamp)newFormatted = UnityAppMod.formatTimestamp(newTimestamp)

# Define the logsource for the searchlogsource = ’"logsources":[{"type":"logSource","name":"/SystemOut"}]’

# Define the output datachartdata = []timeUnit = ’hour’timeUnitFormat = ’MM-dd HH:mm:ss.SSS Z’

timestamp = ’{"timestamp":{"from":"’ + newFormatted + ’","to":"’+ currentFormatted + ’","dateFormat":"MM/dd/yy HH:mm:ss.SSS Z"}}’

# Parameter defaultsstart = 0results = 1

##########################################################def getSearchData(query, facet, sortKey, attributes):

#Search query to be used as content for POST

body = ’{"start":’ + str(start) + ’,"results":’ + str(results) + \’,"filter":{"range":’ + timestamp + ’},’ + \query + ’,’ + \sortKey + ’,’ + \attributes + ’,’ + \facet

#print body

if logsource:body = body + ’,’ + logsource

body = body + ’ }’#print body

# Post the search request


response = unity_connection.post(’/Search’, body, content_type=’application/json; charset=UTF-8’);content = get_response_content(response)

# Convert the response data to JSONdata = {}try:

data = json.loads(content)#print json.dumps(data, sort_keys=False,indent=4,

separators=(’,’, ’: ’))except:

passif ’result’ in data:

result = data[’result’]if ’status’ in result and result[’status’] == ’failure’:

msg = result[’message’]print >> sys.stderr, msgsys.exit(1)

return data##########################################################

##########################################################def getErrorsWarningsVsTime(chartdata):

# Define the query for the searchquery = ’"query":"severity:(W OR E)"’

# Define the facet to be used for the searchfacet = ’"facets":{"dateFacet":{"date_histogram":{"field":"timestamp",

"interval":"’+ timeUnit + ’","outputDateFormat":"’+ timeUnitFormat + ’","nested_facet":{"severityFacet":{"terms":{"field":"severity","size":10}}}}}}’

#Define the sortKey to be used for the search resultssortKey = ’"sortKey":["-timestamp"]’

# get the facetFieldsfacetFields = [

{ "id":"date", "label":"Timestamp", "type":"DATE" },{ "id":"severity", "label":"severity", "type":"TEXT"},{ "id":"count", "label":"Count", "type":"LONG"}

]facetRows = []

# Define the attributesattributes = ’"getAttributes":["timestamp","severity"]’

# do the querydata = getSearchData(query, facet, sortKey, attributes)

if ’facetResults’ in data:

# get the facet resultsfacetResults = data[’facetResults’]#print json.dumps(facetResults, sort_keys=False,indent=4,

separators=(’,’, ’: ’))

if ’dateFacet’ in facetResults:# get the facetRowsdateFacet = facetResults[’dateFacet’]CommonAppMod.dateSort(dateFacet)for dateRow in dateFacet:

for severityRow in dateRow[’nested_facet’][’severityFacet’][’counts’]:

facetRows.append({"date":dateRow[’low’], "severity":severityRow[’term’], "count":severityRow[’count’]} );


#print facetRows

chartdata.append({’id’:’ErrorsWarningsVsTime’,’fields’:facetFields,’rows’:facetRows})

return chartdata

##########################################################

# Define the timestamp for the search#timestamp = ’{"timestamp":{"from":"’ + newFormatted + ’","to":"’

+ currentFormatted + ’","dateFormat":"MM/dd/yy HH:mm:ss.SSS Z"}}’

# Define the logsource for the search#logsource = ’"logsources":[{"type":"logSource","name":"/SystemOut"}]’

getErrorsWarningsVsTime(chartdata)unity_connection.logout()#------------------------------------------------------# Build the final output data JSON#------------------------------------------------------

# build the chart datadt = {’data’:chartdata}

#Print the JSON to system out#print( json.dumps(dt, sort_keys=False,indent=4, separators=(’,’, ’: ’)))print json.dumps(dt, sort_keys=False,indent=4, separators=(’,’, ’: ’))

Application filesThe application file JSON can be created by implementing the structure providedin the sample JSON outlined in this topic.

If you are using the Insight Pack Tooling, the application file JSON can be createdin the Insight Pack project in the subfolder src-files/unity_apps/apps.

The parameters defined for application files are:

name Specify the name of the application file. This is the name of the .app filethat displays in the Custom Search Dashboards pane in the Searchworkspace.

type (Optional) Where applicable, specify the name of the template used. Thename you specify in this field is the name of the template file excludingthe file extension.

descriptionSpecify the purpose of the application.

customLogic Specify the details for the script including any parameters and outline thedetails for the output that is to be displayed. These parameters are definedfor customLogic:

script Specify the name of the script that you want to execute. The scriptcan be a Python or shell script, or a Java™ application packaged asan executable JAR file. If you are going to use a template, the


script you execute must be in the same directory as the template. Ifyou are not using a template, save the script to the same directoryas the application file.

descriptionProvide a description for the script.

timeout(Optional) Specify in seconds the time after which the searchqueries issued by the Custom Search Dashboard are stopped. Thisper-app parameter has no default value.

parametersComplete a JSON array of parameters that are passed to the script.This array can be empty. The parameters you pass depend on therequirements of the script that you are executing. In the first set ofparameters in the sample, the parameters for a search query arepassed.

outputThis section outlines the nature of the output that you want to display foryour Custom Search Dashboard.

type Specify Data as the value for this parameter.

visualizationAdd dashboard or searchFilters as a sub-parameter for thisparameter. This determines what is displayed when you execute aCustom Search Dashboard.

The dashboard requires that you specify the layout for thedashboard that you are creating. Under the Dashboard parameter,add a columns sub-parameter and specify a numeric value toindicate the number of columns that you require.

Add a chart as an additional sub-parameter for the dashboardparameter and add an array of JSON objects to indicate whatcharts you want to include in your dashboard. Include only validchart types and also include the required parameters for eachchart. For more information, see the Charts section of thisdocument.

The searchFilters parameter is for the search filter app andrequires a relationalQuery as the input parameter. The relationalquery is JSON object that consists of JSON keys: For moreinformation see Search filter app.

Example Dashboard visualizations

This example references the ExecuteSearchWithParameters.py script and specifies abubble chart based on data from the chartdata01 data set. The chartdata01 dataset id is defined in the ExecuteSearchWithParameters.py script and referenced inthe chart specification.

"name": "Search Results from python script- Parameters","type": "SearchApps","description": "Captures long running queries in maximo logs","customLogic": {

"script": "ExecuteSearchWithParameters.py","description": "View chart on search results",

"timeout":"parameters": [

{


"name": "search","type": "SearchQuery","value": {"start": 0,"results": 10,

"filter": {"range": {"timestamp": {"from":"12/03/2012 22:21:58.000 India Standard Time",

"to": "12/03/2013 22:21:58.000 India Standard Time","dateFormat": "dd/MM/yyyy HH:mm:ss.SSS Z" }}},

"logsources": [{"type": "tag","name": "*"}],"query": "*"

}},{

"name": "url","type": "String","value": "http: //9.118.41.71: 9988/Unity/"

}],"output": {

"type": "Data","visualization": {

"dashboard": {"columns": 1,"charts": [

{"type": "Bubble Chart","title": "Severity against Time","data": {"$ref": "chartdata01"},"parameters": {"xaxis": "timestamp","yaxis": "severity"}

}]

}} } } }

Related tasks:“Creating the search filter app” on page 57You can create the search filter app to create a customized search query in IBMOperations Analytics - Log Analysis.“Modifying the search filter app core logic” on page 62If you want to extend the default implementation, the core logic of the search filterapp is described here.

Charts:

The chart types specified here are supported by IBM Operations Analytics - LogAnalysis. The chart specifications are contained in the <HOME>/AppsFrameowrk/chartspecs directory.

Displaying a chart: To display a chart, you execute your Custom SearchDashboard from the Search workspace. If you close a chart portlet, you must runthe Custom Search Dashboard again to reopen the chart.

These parameters are defined for all charts:

type Specify the type of chart required. The value must match the ID of a chartspecification that is contained in the <HOME>/AppFramework/chartspecsdirectory. The supported chart types are outlined in this section.

title Specify the chart title.

data Specify the ID for the data element that is represented in the chart. This IDspecified must match to the ID provided in the dashboard specifications.


parametersFields to be displayed in the chart.

Line chart

The line chart is defined with these limitations:v Chart name: Line Chartv Parameters: xaxis and yaxisv Chart Specification:

{"type": "Line Chart","title": "Line Chart ( 2 parameters )","data": {"$ref": "searchResults01"},"parameters": {"xaxis": "timeStamp","yaxis": "throughput"}}

v Aggregation To aggregate the throughput parameter, define the following code:{

"type": "Line Chart","title": "Line Chart ( 2 parameters )","data": {

"$ref": "searchResults01","summarizeData": {

"column": "throughput","function": "sum"

}},"parameters": {

"xaxis": "timeStamp","yaxis": "throughput"

}

The summarizeData key determines whether aggregation is to be performed ornot. column is the name of numeric column that uses the LONG or DOUBLEdata type to perform aggregation on. sum is the aggregation function to beapplied. Supported functions are sum, min, max.

Bar chart

The bar chart is defined with these limitations:v Chart name: Bar Chart1v Parameters: xaxis, yaxis, and categoriesv Limitations: Only integer values are supported for the yaxis parameter.v Chart Specification:

{"type": "Bar Chart","title": "Bar Chart ( 3 parameters )","data": {"$ref": "searchResults01"},"parameters": {"xaxis": "timeStamp",


"yaxis": "CPU","categories": "hostname"}}

Point chart

The point chart is defined with these limitations:v Chart name: Point Chartv Parameters: xaxis and yaxisv Chart Specification:

{"type": "Point Chart","title": "Point Chart ( 2 parameters )","data": {"$ref": "searchResults01"},"parameters": {"xaxis": "timeStamp","yaxis": "errorCode"}}

Pie chart

The pie chart is defined with these limitations:v Chart name: Pie Chartv Parameters: xaxis and yaxisv Chart Specification:

{"type": "Pie Chart","title": "Pie Chart ( 2 parameters )","data": {"$ref": "searchResults03"},"parameters": {"xaxis": "count","yaxis": "severity"}}

Cluster bar chart

The cluster bar chart is defined with these limitations:v Chart name: Cluster Barv Parameters: xaxis, yaxis, and sub-xaxisv Chart Specification:

{"type": "Cluster Bar","title": "Cluster Bar ( 3 parameters )","data": {"$ref": "searchResults02"},"parameters": {"xaxis": "hostname","yaxis": "errorCount","sub-xaxis": "msgClassifier"}}


Bubble chart

The bubble chart is defined with these limitations:v Chart name: Bubble Chartv Parameters: xaxis, yaxis, and categoriesv Chart Specification:

{"type": "Bubble Chart","title": "Bubble Chart ( 3 parameters )","data": {"$ref": "searchResults01"},"parameters": {"xaxis": "timeStamp","yaxis": "CPU","categories": "errorCode"}}

The size of the bubble on the graph depends on the number of items in theparameter that is being represented. In some cases, for example if you have alarge bubble and a small bubble, the large bubble may cover the smaller one.

Tree map chart

The tree map chart is defined with these limitations:v Chart name: Tree Mapv Parameters: level1, level2, level3, and valuev Chart Specification:

{"type": "Tree Map","title": "Tree Map ( 4 parameters )","data": {"$ref": "searchResults01"},"parameters": {"level1": "hostname","level2": "errorCode","level3": "severity","value":"CPU"

}}

Two-series line chart

The two-series line chart is defined with these limitations:v Chart name: Two Series Line Chartv Parameters: xaxis, yaxis1, and yaxis2v Chart Specification:

{"type": "Two Series Line Chart","title": "Two Series Line Chart ( 3 parameters)","data": {"$ref": "searchResults01"},"parameters": {"xaxis": "timeStamp",


"yaxis1": "throughput","yaxis2": "ResponseTime"}}

Stacked bar chart

The stacked bar chart is defined with these limitations:v Chart name: Stacked Bar Chartv Parameters: xaxis, yaxis, and categoriesv Chart Specification:

{"type": "Stacked Bar Chart","title": "Stacked Bar Chart ( 3 parameters )","data": {"$ref": "searchResults01"},"parameters": {"xaxis": "hostname","yaxis": "CPU","categories": "severity"}}

Stacked line chart

The stacked line chart is defined with these limitations:v Chart name: Stacked Line Chartv Parameters: xaxis, yaxis, and categoriesv Chart Specification:

{"type": "Stacked Line Chart","title": "Stacked Line Chart ( 3 parameters )","data": {"$ref": "searchResults01"},"parameters": {"xaxis": "threadID","yaxis": "timestamp","categories": "MBO"}}

Heat map

The heat map chart is defined with these limitations:v Chart name: Heat mapv Parameters: xaxis, yaxis, and countv Chart Specification:

{"type": "Heat Map","title": "Heat Map ( 3 parameters )","data": {

"$ref": "searchResults01"},"parameters": {"xaxis": "messageClassifier",


"yaxis": "hostname","count": "throughput"}

}

Example application file:

This example shows an application file that references the3Params_was_systemout.py script example and specifies how multiple chartsdisplay the output JSON from the script.

Example application file and chart dashboard

This example shows the application file and the dashboard of charts it createswhen you run the application. The ErrorsWarningsVsTime and ExceptionByHostdata sets are defined in the 3Params_was_systemout.py script and referenced in thechart specifications in this application file.{"name": "WAS Errors and Warnings Dashboard","description": "Display a dashboard of charts that show WAS errors and warnings","customLogic": {

"script": "3Params_was_systemout.py","description": "View charts based on search results","parameters": [

{"name": "search","type": "SearchQuery","value": {

"logsources": [{

"type": "logSource","name": "SystemOut"

}]

}},{"name": "relativeTimeInterval","type": "string","value":"LastDay"

},{"name": "timeFormat","type": "data","value": {

"timeUnit": "hour","timeUnitFormat": "MM-dd HH:mm"

}},{"name": "hostnameField","type": "string","value": "logsource_hostname"

}],"output": {



{"title": "Message Counts - Top 5 over Last Day","parameters": {


"xaxis": "date","yaxis": "count","sub-xaxis": "severity"},"data": {"$ref": "ErrorsWarningsVsTime"},"type": "Cluster Bar"

},{

"type": "Heat Map","title": "Message Counts - Top 5 over Last Day","data": {"$ref": "ErrorsWarningsVsTime"

},"parameters": {

"xaxis": "date","yaxis": "count","category": "severity"

}},{

"type": "Bubble Chart","title": "Java Exception Counts - Top 5 over Last Day","data": {

"$ref": "ErrorsWarningsVsTime"},"parameters": {

"xaxis": "date","yaxis": "count","size": "severity"

}

},{

"type": "Two Series Line Chart","title": "Error and Warning Total by Hostname

- Top 5 over Last Day","data": {


"xaxis": "date","yaxis1": "count","yaxis2": "severity"

}

},{

"type": "Tree Map","title": "Messages by Hostname - Top 5 over Last Day","data": {


"level1": "date","level2": "count","level3": "severity","value": "count"

}

},{

"type": "Stacked Bar Chart","title": "Java Exception by Hostname

- Top 5 over Last Day","data": {


"$ref": "ExceptionByHost"},"parameters": {

"xaxis": "date","yaxis": "count","categories": "hostname"

}

}

]}

}}

}

}

This sample shows a chart created from the example script and application files.

Using a Custom Search Dashboard to display a search query or selected data:

You can create a Custom Search Dashboard that displays, in a new tab, the fullsyntax for the current search and also displays the contents of a selected column orindividual cell.

Create an application and include the _query and _data parameters to ensure thatyour data is displayed. The purpose of these parameters is:

_queryDisplays a JSON string containing the currently active search. Specify_query in the name field for the parameter. The type field must contain thevalue additionalParameterFromUI.

_data Displays the data from any column or cell that you select in Grid view. Todisplay the data, select a column or cell in Grid view and then launch the


application. If you select a column, only data from the currently displayedpage is displayed by the application. Specify _data in the name field for theparameter. The type field must contain the valueadditionalParameterFromUI.

_rows Displays the entire row data for a cell that you select in Grid view. Todisplay the data, select a cell in Grid view and then launch the application.The type field must contain the value additionalParameterFromUI.

The application does not necessarily require all three parameters. Depending onyour requirements, you can create an application to display the search query JSONor the contents of a column, cell, or row.

The following example is an application that when executed displays both thesearch query and any selected column or cell:{

"name": "Sample HTML App","description": "Sample App to demonstrate use of _query and _data params","customLogic": {

"script": "SampleHTMLApp.sh","description": "Read data from a file and return","parameters": [

{"name": "_query","type": "additionalParameterFromUI","value": []

},{

"name": "_data","type": "additionalParameterFromUI","value": []

},{

"name": "_rows","type": "additionalParameterFromUI","value": []

}],"output": {



{"type": "html","title": "Sample HTML","data": {

"$ref": "htmlData"},"parameters": {

"html": "text"}

}]

}}

}}

}

Related concepts:


“Launching IBM Operations Analytics - Log Analysis with context parameters”You can launch IBM Operations Analytics - Log Analysis with search parametersor Custom Search Dashboard in the URL that you specify. These URLs can bespecified and launched from within applications that are integrated with IBMOperations Analytics - Log Analysis.

Launching IBM Operations Analytics - Log Analysis with context parameters:

You can launch IBM Operations Analytics - Log Analysis with search parametersor Custom Search Dashboard in the URL that you specify. These URLs can bespecified and launched from within applications that are integrated with IBMOperations Analytics - Log Analysis.

Launching with search parameters in the URL

The URL format that you specify must be in the format:https://<ip_address>:9987:/Unity/SearchUI?queryString=<q>&timefilters=<t>&dataSources=<ds>

where <ip_address> is the IP address of the server on which IBM OperationsAnalytics - Log Analysis is installed and the parameters that you specify are:

queryString(Required) Replace the <q> parameter with a valid velocity query.

timefilter(Optional) Specify a time range as an absolute or relative time range inJSON format. If this parameter is not specified, the default option Last 15minutes is applied. For an absolute value, specify JSON using thisexample:{ “type”:“absolute”,

“startTime”:“24/06/2013 05:30:00”“endTime”:“25/06/2013 05:30:00”}

The timezone for the startTime and endTime is the user's timezone. If theuser's default timezone is the browser timezone, the absolute time is thebrowser timezone or the timezone that is set as default for all sessions isused for querying.

For a relative time range, specify JSON using this example:{ “type”:“relative”,

“lastnum”:“7”,“granularity”:“Day”}

datasources(Optional) For this parameter, you specify a Data Source or a group ofData Sources in a JSON array format. Each element in the array can be oftype group or datasource. Specify group if you want to specify a group ofData Sources. If a value is not specified for this parameter, all of theavailable Data Sources are selected. The JSON format is indicated in thisexample:[

{ “type”:“datasource”,“name”:"<datasource_name>" },{ “type”:“group”,“name”:"<group_name>}", ...

]


This example URL launches IBM Operations Analytics - Log Analysis with therequired parameters to search for critical IBM Tivoli Netcool/OMNIbus events inIBM Operations Analytics - Log Analysis:https://9.118.41.69:9987/Unity/SearchUI?queryString=*&timefilters={“type”:“relative”,“lastnum”:1,“granularity”:“year”}&dataSources=[{“type”:“datasource”,“name”:/Omnibus-events}]

Note: Any spaces that are required in your URL must be escaped. If you want toinclude a % character in your URL in must be added as %25.

Launching with a Custom Search Dashboard name in a URL

The URL format that you specify must be in the format:https://<ip_address>:9987:/Unity/CustomAppsUI?name=<name>&<appParameters>=<params>

where <ip_address> is the IP address of the server on which IBM OperationsAnalytics - Log Analysis is installed and the parameters that you specify are:

name (Required) Replace the <name> parameter with a valid Custom SearchDashboard name.

appParametersIf the Custom Search Dashboard that you want to launch requires thatparameters are specified, specify the required parameters in a JSON arrayin {key:value} format.

This example URL launches the IBM Operations Analytics - Log Analysis DayTrader Custom Search Dashboard:https://9.120.98.21:9987/Unity/CustomAppsUI?name=Day%20Trader%20App&appParameters=[]

Note: Any spaces that are required in your URL must be escaped. If you want toinclude a % character in your URL, it must be added as %25.Related concepts:“Using a Custom Search Dashboard to display a search query or selected data” onpage 45You can create a Custom Search Dashboard that displays, in a new tab, the fullsyntax for the current search and also displays the contents of a selected column orindividual cell.

TemplatesA template file defines a set of custom scripts. For each custom script, it specifiesthe parameters with the type, name, and the default value.

If you are using the Insight Pack Tooling, the template file JSON can be created inan Insight Pack project in the subfolder src-files/unity_apps/templates. Anyscript files referenced by the template must also reside in this folder.

In addition to the fields included for applications files, these additional parametersare included in the template:

templateThis is a boolean value that specifies that the file is a template. Specifytrue as the value for this parameter.


parameterAlthough this parameter is specified in the application file, additionalvalues are required in a template file for this parameter.

requiredIf this is set to true, the parameter is required. Custom SearchDashboards that use the template must include this parameter.

defaultSpecify the default parameter value. For parameters that are notrequired, this value is used where the application does not specifya value.

Example{

"name": "SearchApps","type": "SearchApps","template": true,"description": "SearchApps Template","customLogic": [

{"script": "ExecuteSearch.py","description": "View chart on search results","parameters": [],"output": {


"dashboard": {"charts": [{"type": "Pie Chart","title":"Errors/Warnings for last 7 days","yaxis": "msgclassifier"},

{"type": "Bar Chart","title":"Errors/Warnings for each host",

"xaxis": "hostname","yaxis": "count"}]}} } },{

"script": "ExecuteSearchWithParameters.py","description": "View chart on search results","parameters": [

{"name": "search","type": "SearchQuery","default": {"start": 0,"results": 10,"filter": {

"range": {"timestamp": {"from": "12/03/2012 22:21:58.000India Standard Time",

"to": "12/03/2013 22:21:58.000 India Standard Time","dateFormat": "dd/MM/yyyy HH:mm:ss.SSS Z"}}},

"logsources": [{"type": "tag","name": "*"}],"query": "*"},

"required": false},{

"name": "url","type": "String","default": "http: //9.118.41.71: 9988/Unity/","required": false

}],"output": {


"dashboard": {"charts": [

{"type": "Pie Chart","title": "Errors/Warnings forlast 7 days","yaxis": "msgclassifier"},

{"type": "Bar Chart","title": "Errors/Warnings for


each host","xaxis": "hostname","yaxis": "count"}]}}

}}

]}

Building and Installing an Insight Pack with Custom SearchDashboardsThe Insight Pack Tooling creates an archive file for the Insight Pack using the BuildInsight Pack option from the project's context menu.

Before you begin

The script and application definition files must be created in the Insight Packproject subfolder src-files/unity_apps/apps. If you are using a template, thescript, application definition, and template files must be created in the Insight Packproject subfolder src-files/unity_apps/templates.

Procedurev Any Custom Search Dashboard definitions, scripts, and template files found in

the src-files/unity_apps folder are included in the archive file.v When the Insight Pack archive file is installed using the pkg_mgmt utility, the

application definition, scripts, and template files are installed to the appropriatedirectories under <HOME>/AppFramework. You may need to refresh the CustomApps pane in the Search workspace in order to display newly installed CustomSearch Dashboards.

Example of full Custom Search DashboardThis example shows a complete script, application file, and the resulting displayfor a Custom Search Dashboard named Performance_Msgs.

Performance_Msgs.py script

This example shows the script Performance_Msgs.py, which collects data onperformance messages. For details, read the coding notes within the script.######################################################### {COPYRIGHT-TOP} #### Licensed Materials - Property of IBM# "Restricted Materials of IBM"# 5725-K26## (C) Copyright IBM Corp. 2013 All Rights Reserved.## US Government Users Restricted Rights - Use, duplication, or# disclosure restricted by GSA ADP Schedule Contract with IBM Corp.######################################################### {COPYRIGHT-END} ####Install simple json before trying the script

#Installing simplejson# Download the rpm fromhttp://pkgs.org/centos-5-rhel-5/centos-rhel-x86_64/python-simplejson-2.0.9-8.el5.x86_64.rpm/download/# Run the command rpm -i python-simplejson-2.0.9-8.el5.x86_64.rpm

from datetime import datetime, date, time, timedeltaimport sysfrom unity import UnityConnection, get_response_contenttry:

import json


except ImportError:import simplejson as json

#------------------------------------------------------# getSearchData()#------------------------------------------------------def getSearchData(logsource, filter):

# Build the request parameters using the Search Request APIs.# The logsource and filter values used in the query were passed# in as parameters to the script.request = {

"start": 0,"results": 1,"filter": filter,"logsources": logsource,"query": "*","sortKey":["-timestamp"],"getAttributes":["timestamp","perfMsgId"],"facets":{

"dateFacet":{"date_histogram":{

"field":"timestamp","interval":"hour","outputDateFormat":"MM-dd HH:mm","nested_facet":{

"dlFacet":{"terms":{

"field":"perfMsgId","size":20

}}

}}

}}

}

# Post the search requestresponse = connection.post(’/Search’, json.dumps(request), content_type=’application/json;charset=UTF-8’);

content = get_response_content(response)

#convert the response data to JSONdata = {}try:

data = json.loads(content)except:

passif ’result’ in data:

result = data[’result’]if ’status’ in result and result[’status’] == ’failure’:msg = result[’message’]print >> sys.stderr, msgsys.exit(1)

return data

#------------------------------------------------------# dateSort()#------------------------------------------------------def dateSort(dateFacet):

# This function parses the UTC label found in the dateFacet in the# format "mm-hh-DDD-yyyy UTC"# and returns an array in the form [yyyy, DD, hh, mm]def parseDate(dateLabel):


aDate = map(int, dateLabel.split(" ")[0].split("-"))aDate.reverse()return aDate

# call an in-place List sort, using an anonymous function lambda# as the sort functiondateFacet.sort(lambda facet1, facet2:cmp(parseDate(facet1[’label’]), parseDate(facet2[’label’])))return dateFacet

#--------------------------------------------------------------------------# Main script starts here#--------------------------------------------------------------------------

# define the URLs used for the http requestbaseurl = ’https://localhost:9987/Unity’

connection = UnityConnection(baseurl, ’unityadmin’, ’unityadmin’)connection.login()

# initialize variablesfilter = {}logsource = {}chartdata = []

# Get the script parameters which were passed in via a temporary# file in JSON format. The name of the temporary file is in argv[1]if len(sys.argv) > 1:

filename = str(sys.argv[1])fk = open(filename,"r")data = json.load(fk)

parameters = data[’parameters’]for i in parameters:

if i[’name’] == ’search’:search = i[’value’]for key in search.keys():

if key == ’filter’:filter = search[’filter’]

elif key == ’logsources’:logsource = search[’logsources’]

#------------------------------------------------------# get the data to be returned#------------------------------------------------------

# define the fields to be returned in the chart data# each row will contain the date, perfMsgId, and countfields = [

{ "id":"date", "label":"Timestamp", "type":"TEXT"},{ "id":"perfMsgId", "label":"perfMsgId", "type":"TEXT" },{ "id":"count", "label":"count", "type":"LONG"}

]

rows = []

# call getSearchData() to post the search request and retrieve the datadata = getSearchData(logsource, filter)

if ’facetResults’ in data:

# get the facet resultsfacetResults = data[’facetResults’]


#printjson.dumps(facetResults, sort_keys=False,indent=4, separators=(’,’, ’: ’))

if ’dateFacet’ in facetResults:# get the dateFacet rowsdateFacet = facetResults[’dateFacet’]

# the results of the dateFacet are not sorted, so call dateSort()dateSort(dateFacet)

# iterate through each row in the dateFacetfor dateRow in dateFacet:

for msgRow in dateRow[’nested_facet’][’dlFacet’][’counts’]:# create a row which includes the date, perfMsgId, and countrows.append({"date":dateRow[’low’], "perfMsgId":msgRow[’term’],"count":msgRow[’count’]} );#print rows

# create chart data with the id perfMsgIdCountsOverTimechartdata.append({’id’:’perfMsgIdCountsOverTime’,’fields’:fields, ’rows’:rows})

# close the connectionconnection.logout()

#------------------------------------------------------# Create the HTML data to be returned#------------------------------------------------------html = "<!DOCTYPE html><html><body><h1>Hello World!</h1></body></html>"</p><p>chartdata.append({"id": "htmlData", "htmltext": html })

#------------------------------------------------------# Build the final output data JSON#------------------------------------------------------

# build the JSON structure containing the chart data which will be the# output of this scriptappData = {’data’:chartdata}

#Print the JSON to system outprint json.dumps(appData, sort_keys=False,indent=4, separators=(’,’, ’: ’))

Performance_Msgs.app file

This sample shows the Performance_Msgs.app application file that references thePerformance_Msgs.py script and specifies the chart to display for the CustomSearch Dashboard.{

"name": "Performance Messages","description": "Displays charts showing performance messages over time","customLogic": {

"script": "Performance_Msgs.py","description": "View chart on search results","parameters": [

{"name": "search","type": "SearchQuery","value": {

"filter": {"range": {

"timestamp":{"from":"01/01/2013 00:00:00.000 EST","to":"01/01/2014 00:00:00.000 EST","dateFormat":"MM/dd/yyyy HH:mm:ss.SSS Z"

}}

},


"logsources": [{

"type": "logSource","name": "MyTest"

}],

}},

],"output": {



{"type": "Stacked Bar Chart","title": "Performance Message ID Counts - Last Year","data": {

"$ref": "perfMsgIdCountsOverTime"},"parameters": {

"xaxis": "date","yaxis": "count","categories": "perfMsgId"

}},{

"type": "html","title": "My Test - Expert Advice","data": {

"$ref": "htmlData"},"parameters": {

"html": "text"}

}

]}

}}

}}


Performance_Msgs chart

The chart shows the counts of each message for each day.

Defining a search filter appYou can use the search filter app to create a customized search query in IBMOperations Analytics - Log Analysis.

A search filter app is a custom search request that:v searches databases for strings and timestampsv creates Configured Patterns from the found stringsv uses the minimum and maximum timestamps found in the database to frame

the search time range.

Before you create the search filter application, define the database connections. Formore information, see “Database Connections” on page 56.

IBM Operations Analytics - Log Analysis includes two files in the<HOME>/IBM/LogAnalysis/AppFramework/Templates/SearchFilters directory thatyou can use to implement a customized search filter app:v searchFilter.jar is self executable JAR file that contains Java classes. The

classes support the core functions of the app.v searchFilter.sh is a wrapper script that starts the searchFilter.jar file after

the appropriate class path is set.

You must copy these files to the directory in <HOME>/IBM/LogAnalysis/AppFramework/Apps that contains your application (.app) file.

Supported databases

The following databases are supported by defaultv Derby 10.10v DB2 9.7


To add databases that are not supported by default, you must locate the JAR filethat contains the class 4 JDBC driver for the database, change the location of thedrivers in the classpath section of the searchFilter.sh file.1. Download the appropriate class 4 JDBC driver file for your database.2. Copy the driver file to the <HOME>/IBM/LogAnalysis/AppFramework/Apps/

<directory> directory where <directory> is the directory that contains yoursearch filter application.

3. Add the location of the new driver to the classpath parameter in thesearchFilter.sh file.

At time of publication, Oracle Database 11g Release 11.1.0.0.0 is the only additionaldatabase tested.

Database ConnectionsCreate a database connection to help your search filter application to uniquelyidentifies sources of structured data.

You can associate semi-structured data, such as a log file, with structured data,such as the transaction status in the application database. To make this association,you must define a Database Connection, which specifies the details required byIBM Operations Analytics - Log Analysis to access the database. After you havedefined a Database Connection, you can use it when you configure your searchfilter application.

Adding a Database Connection:

This topic outlines the steps that you must follow to configure an events databaseas a Database Connection.

Procedure

To add a Database Connection:1. In the Data Sources workspace, click Add > Database Connections. The Add

Database Connections tab is displayed2. You are prompted to supply the following information:

Name The name of the Database Connection.

Schema NameThe name of the table containing the event, for example, MISC.

JDBC URLThe IP address and path for the events database, for example,jdbc:derby://ip_address:1627//opt/unity/database/UnityDB.

JDBC DriverThe JDBC driver, typically org.apache.derby.jdbc.ClientDriver.

UsernameType the username for the Database Connection.

PasswordType the password for the Database Connection.

3. Click OK. A new entry is displayed in the Database Connections list.

Editing a Database Connection:

You can edit the details of an existing Database Connection.


Procedure

To edit a Database Connection:1. In the Data Sources workspace, expand the Database Connections list.2. Select the Database Connection that you want to edit and click Edit. The

Database Connection is opened in a new tab.3. Edit the data source as required.4. To save the changes, click OK.

Deleting a Database Connection:

You can delete an existing Database Connection.

Procedure

To delete a Database Connection:1. In the Data Sources workspace, expand the Database Connections list.2. Select the Database Connection that you want to delete and click Delete.3. Click OK

Creating the search filter appYou can create the search filter app to create a customized search query in IBMOperations Analytics - Log Analysis.

About this task

To create a search filter app:

Procedure1. SearchFilter definition: Use this app definition to define your own search filter:

{"name": "Example app for search filter","description": "Example app for search filter","customLogic": {"script": "searchFilter.sh","description": "Example app for search filter","parameters": [

{"name": "relationalQuery","type": "relationalQuery","value": {

"dataSource": {DATASOURCE_DETAILS},"SQL": "SQL_QUERY"}

}],

"output": {"type": "searchFilters","visualization": {"searchFilters": {}}}}}

2. Update the dataSource field with your datasource details.For example:


"value": {"dataSource": {"schema": "ITMUSER","jdbcdriver": "com.ibm.db2.jcc.DB2Driver","jdbcurl": "jdbc:db2://9.12.34.56:50000/WALE","datasource_pk": 1,"username": "itmuser","password": "tbsm4120","name": "datasource"},

3. Update the SQL field with your query.For example:"SQL": "select * from ITMUSER.test_OLSC"

Results

After you run the app, the customized search is displayed in the configuredpattern section of the user interface and the data source and time filters are set.The keywords, if any, and the count information are also displayed.

Note:

1. If the app output contains data sources, the values that match the searchcriteria are returned on the user interface. If no data sources are returned, theexisting selections are retained and used.

2. If the app output contains time filters, the values that match the search criteriaare returned on the UI. If no time filters are returned, the existing selections areretained and used.

3. If the app output contains keywords, IBM Operations Analytics searches thedata sources and time filters that are returned and displays the keywords andthe search hit count in the configured patterns widget UI.

Example

The following code example demonstrates the logic that is used for the search filterapp:{"name": "Sample Search filter app","description": "App for getting search filter","customLogic": {"script": "searchFilter.sh","description": "App for getting search filter","parameters": [{"name": "relationalQuery","type": "relationalQuery","value": {"dataSource": {"schema": "ITMUSER","jdbcdriver": "com.ibm.db2.jcc.DB2Driver","jdbcurl": "jdbc:db2://9.12.34.56u:50000/WALE","datasource_pk": 1,"username": "itmuser","password": "tbsm4120","name": "datasource"

},"SQL": "select * from ITMUSER.test_OLSC"}}],"output": {


"type": "searchFilters","visualization": {"searchFilters": {}}}}}

Note: The output > visualization sub parameter must be set to searchFilters.

The search filter app uses relationalQuery as the input parameters. The relationalquery is a JSON object that consists of the following JSON keys:v dataSource is the value of the data source key that defines database connection

attributes such as JDBC connection details, schema name, user credentials .v SQL is the value of SQL key that defines the query that we want to run against

the database

Note: If you are trying to run search filter against a data source registered withIBM Operations Analytics, you can provide dataSource name instead of dataSourcedetails. All details of the corresponding dataSource are fetched from IBMOperations Analytics. The dataSource is created with the Database connectionsoption in the Data Sources workspace.{

"name": "relationalQuery","type": "relationalQuery","value": {"dataSource": "myDataSourceName","SQL": "select * from MYDB.MYTABLE"}

}

where myDataSourceName is the name of a data source that is defined in IBMOperations Analytics - Log Analysis.Related concepts:“Application files” on page 36The application file JSON can be created by implementing the structure providedin the sample JSON outlined in this topic.

Data elements:

There are three parameters required for data elements:

id Specify an identifier for the data element. The Custom Search Dashboarduses this value to determine the data that is displayed. The Custom SearchDashboard uses this value to determine the data that is displayed for aparticular chart.

fields Specify an array containing field descriptions. Each field that you specifyin the array has an ID, a label, and a type.

rows Using the rows array to specify a value for each of the fields that you havespecified.

HTML elements:

There are two parameters required for HTML elements:

id Specify an identifier for the data element. The Custom Search Dashboard


uses this value to determine the data that is displayed. In the applicationfile, the chart specifications use this id to specify the data element used forthe chart.

htmltextSpecify the HTML that you want to display. A HTML portlet is used todisplay the HTML that you specify.

Example

This is a sample JSON output which contains both the types of output whichcontains both chart data and HTML output:{

"data":[{

"id":"ErrorsWarningsVsTime","fields":[

{"id":"timestamp","label":"Date","type":"TEXT"

},{

"id":"severity","label":"severity","type":"TEXT"

},{

"id":"count","label":"count","type":"LONG"

}],"rows":[

{"timestamp":"10-21 13:00","severity":"W","count": 221

},{

"timestamp":"10-21 13:00","severity":"E","count": 204

}]

},{

"id":"htmlData","htmltext":",<!DOCTYPE html><html><body><div><h1>Sample HTML</h1></div></body></html>"

}]

}

In this example, the data id ErrorsWarningsVsTime is defined in the script.chartdata.append({’id’:’ErrorsWarningsVsTime’,’fields’:facetFields,’rows’:facetRows})

In the application file chart specification, the id ErrorsWarningsVsTime is referencedto specify the data set used in the chart.


{"type": "Tree Map","title": "Messages by Hostname - Top 5 over Last Day","data": {


"level1": "date","level2": "count","level3": "severity","value": "count"

}

Launching a new context tab in a HTML Custom Search Dashboard

You can provide a link within a HTML Custom Search Dashboard that launches aURL in a new tab within the Custom Search Dashboard. The search or CustomSearch Dashboard is executed with the context parameters that you specify in theURL.

This example contains the URLs to launch searches with search parameters withina Custom Search Dashboard:{"id": "htmlData","htmltext": "<!DOCTYPE html><html><body><br><div><a href=http://www.espncricinfo.com//>Normal URL -- ESPN Cricinfo</a></div><br><div><a href=https://192.168.56.101:9987/Unity/CustomAppsUI?name=All%20Supported%20Visualizations&appParameters=[]>Custom App LIC- - All Supported Visualizations</a></div><br><div><a href=https://192.168.56.101:9987/Unity/SearchUI?queryString=diagnosticLevel:==Warning&timefilters={\"type\":\"relative\",\"lastnum\":\"1\",\"granularity\":\"year\"}&dataSources=[{\"type\":\"datasource\",\"name\":\"/DB2Diag1\"}]>Search UI LIC - - DB Log Events with Diagnostic Level - Warning</a></div><br><div><a href=https://192.168.56.101:9987/Unity/SearchUI?queryString=diagnosticLevel:==Severe&timefilters={\"type\":\"relative\",\"lastnum\":\"1\",\"granularity\":\"year\"}&dataSources=[{\"type\":\"datasource\",\"name\":\"/DB2Diag1\"}]>Search UI LIC - - DB Log Events with Diagnostic Level - Severe</a></div></body></html>" }

Note: Any spaces that are required in your URL must be escaped. If you want toinclude a % character in your URL in must be added as %25.

Template-based search filter Custom Search Dashboard:

A template has been provided for search filter Custom Search Dashboards. Thistemplate is located in the <HOME>/IBM/LogAnalysis/AppFramework/Templates/SearchFilters directory.

To use this template, you must specify the template name in the type field:{"name": "Sample Search filter app","type":"SearchFiltersTemplate","description": "App for getting search filter","customLogic": {"script": "searchFilter.sh","description": "App for getting search filter","parameters": [{"name": "relationalQuery","type": "relationalQuery","value": {


"dataSource": {"schema": "ITMUSER","jdbcdriver": "com.ibm.db2.jcc.DB2Driver","jdbcurl": "jdbc:db2://9.12.34.56u:50000/WALE","datasource_pk": 1,"username": "itmuser","password": "tbsm4120","name": "datasource"

},"SQL": "select * from ITMUSER.test_OLSC"}}],"output": {"type": "searchFilters","visualization": {"searchFilters": {}}}}}

If you use this template, the application directory for this application requires theapplication file only. You do not need to copy the .sh and JAR script files into theapplication folder.

Modifying the search filter app core logicIf you want to extend the default implementation, the core logic of the search filterapp is described here.

Procedure1. Create a connection to the datasource or database that you specified in the

input file using the Data Sources workspace.2. Run the SQL query that is part of the input file for the app.3. Tokenize the output of the SQL query and remove common words such as

verbs.4. Pick out the timestamp column and pick up startTime as minimum timestamp

and endTime as maximum timestamp.5. If you already know the data source against which you want to use this search

filter, you can use the logsource name in the output parameters. If you do notknow this, use an asterisk (*).

6. Construct output in the JSON format. For example:{

"keywords": ["keyword1","keyword2",],"timefilters": {"type": "absolute","startTime":"2012-06-24 05:30:00","endTime":"2013-06-24 05:30:00","lastnum": 7,"granularity": "day"},"logsources": [{"type": "tag","name": "*"}]}


Note: The source code of the default implementation (in Java) of the searchfilter app is in included with IBM Operations Analytics - Log Analysis in the<HOME>/AppFramework/Templates/SearchFilters directory. You can update thedefault implementation based on your requirements.

Related concepts:“Application files” on page 36The application file JSON can be created by implementing the structure providedin the sample JSON outlined in this topic.

Adding a shortcut to a Custom Search Dashboard to the Tableview toolbar

You can add a Custom Search Dashboard to the Table view toolbar so that you canquickly launch your Custom Search Dashboard.

Procedure1. Open the CustomAppsConfigFile.json Custom Search Dashboard configuration

file located in the <HOME>/wlp/usr/servers/Unity/apps/Unity.war/configsdirectory.

2. Add a link, an icon image, and a tooltip for each Custom Search Dashboard forwhich you want to create a shortcut. The syntax for the shortcut is:[{"url": "<url>","icon": "<icon>","tooltip": "<tooltip>"

}]

where:

url (Required)

The URL format that you specify must be in the format:https://<ip_address>:9987:/Unity/CustomAppsUI?name=<name>&<appParameters>=<params>

where <ip_address> is the IP address of the server on which IBMOperations Analytics - Log Analysis is installed and the parameters thatyou specify are:

name (Required) Replace the <name> parameter with a valid CustomSearch Dashboard name.

appParametersIf the Custom Search Dashboard that you want to launchrequires that parameters are specified, specify the requiredparameters in a JSON array in {key:value} format.

icon Specify the path to the graphic that you want to use for your icon. If noicon is specified, the default Custom Search Dashboard icon is used.

tooltipSpecify the text for the tooltip displayed when the mouse is placed onthe icon. If no tooltip is specified, the name of the Custom SearchDashboard is used as the tooltip text.

This example provides a link to a Custom Search Dashboard named AnomalyAppwith a path to a shortcut icon located in the Unity/images directory and atooltip text Detect Anomalies:


[{"url": "https://192.168.56.101:9987/Unity/CustomAppsUI?name=Anomaly

App&appParameters=[]","icon": "https://192.168.56.101:9987/Unity/images/context.gif","tooltip": "Detect Anomalies"

}]

Note: Any spaces that are required in your URL must be escaped. If you wantto include a % character in your URL in must be added as %25.

3. Save the file.

Results

Your shortcut is added to the Table view toolbar. Double-click the icon to launchyour Custom Search Dashboard.


Notices

This information was developed for products and services that are offered in theUSA.

IBM may not offer the products, services, or features discussed in this document inother countries. Consult your local IBM representative for information on theproducts and services currently available in your area. Any reference to an IBMproduct, program, or service is not intended to state or imply that only that IBMproduct, program, or service may be used. Any functionally equivalent product,program, or service that does not infringe any IBM intellectual property right maybe used instead. However, it is the user's responsibility to evaluate and verify theoperation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matterdescribed in this document. The furnishing of this document does not grant youany license to these patents. You can send license inquiries, in writing, to:

IBM Director of LicensingIBM CorporationNorth Castle Drive, MD-NC119Armonk, NY 10504-1785United States of America

For license inquiries regarding double-byte character set (DBCS) information,contact the IBM Intellectual Property Department in your country or sendinquiries, in writing, to:

Intellectual Property LicensingLegal and Intellectual Property LawIBM Japan Ltd.19-21, Nihonbashi-Hakozakicho, Chuo-kuTokyo 103-8510, Japan

The following paragraph does not apply to the United Kingdom or any othercountry where such provisions are inconsistent with local law:INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THISPUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHEREXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIEDWARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESSFOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express orimplied warranties in certain transactions, therefore, this statement may not applyto you.

This information could include technical inaccuracies or typographical errors.Changes are periodically made to the information herein; these changes will beincorporated in new editions of the publication. IBM may make improvementsand/or changes in the product(s) and/or the program(s) described in thispublication at any time without notice.

Any references in this information to non-IBM websites are provided forconvenience only and do not in any manner serve as an endorsement of those


websites. The materials at those websites are not part of the materials for this IBMproduct and use of those websites is at your own risk.

IBM may use or distribute any of the information you supply in any way itbelieves appropriate without incurring any obligation to you.

Licensees of this program who wish to have information about it for the purposeof enabling: (i) the exchange of information between independently createdprograms and other programs (including this one) and (ii) the mutual use of theinformation which has been exchanged, should contact:

IBM Corporation2Z4A/10111400 Burnet RoadAustin, TX 78758 U.S.A.

Such information may be available, subject to appropriate terms and conditions,including in some cases, payment of a fee.

The licensed program described in this document and all licensed materialavailable for it are provided by IBM under terms of the IBM Customer Agreement,IBM International Program License Agreement or any equivalent agreementbetween us.

Any performance data contained herein was determined in a controlledenvironment. Therefore, the results obtained in other operating environments mayvary significantly. Some measurements may have been made on development-levelsystems and there is no guarantee that these measurements will be the same ongenerally available systems. Furthermore, some measurements may have beenestimated through extrapolation. Actual results may vary. Users of this documentshould verify the applicable data for their specific environment.

Information concerning non-IBM products was obtained from the suppliers ofthose products, their published announcements or other publicly available sources.IBM has not tested those products and cannot confirm the accuracy ofperformance, compatibility or any other claims related to non-IBM products.Questions on the capabilities of non-IBM products should be addressed to thesuppliers of those products.

All statements regarding IBM's future direction or intent are subject to change orwithdrawal without notice, and represent goals and objectives only.

All IBM prices shown are IBM's suggested retail prices, are current and are subjectto change without notice. Dealer prices may vary.

This information is for planning purposes only. The information herein is subject tochange before the products described become available.

This information contains examples of data and reports used in daily businessoperations. To illustrate them as completely as possible, the examples include thenames of individuals, companies, brands, and products. All of these names arefictitious and any similarity to the names and addresses used by an actual businessenterprise is entirely coincidental.

COPYRIGHT LICENSE:


This information contains sample application programs in source language, whichillustrate programming techniques on various operating platforms. You may copy,modify, and distribute these sample programs in any form without payment toIBM, for the purposes of developing, using, marketing or distributing applicationprograms conforming to the application programming interface for the operatingplatform for which the sample programs are written. These examples have notbeen thoroughly tested under all conditions. IBM, therefore, cannot guarantee orimply reliability, serviceability, or function of these programs. The sampleprograms are provided "AS IS", without warranty of any kind. IBM shall not beliable for any damages arising out of your use of the sample programs.

© Copyright IBM Corp. 2015. All rights reserved.

TrademarksIBM, the IBM logo, and ibm.com are trademarks or registered trademarks ofInternational Business Machines Corp., registered in many jurisdictions worldwide.Other product and service names might be trademarks of IBM or other companies.A current list of IBM trademarks is available on the web at www.ibm.com/legal/copytrade.shtml.

Terms and conditions for product documentationPermissions for the use of these publications are granted subject to the followingterms and conditions.

Applicability

These terms and conditions are in addition to any terms of use for the IBMwebsite.

Personal use

You may reproduce these publications for your personal, noncommercial useprovided that all proprietary notices are preserved. You may not distribute, displayor make derivative work of these publications, or any portion thereof, without theexpress consent of IBM.

Commercial use

You may reproduce, distribute and display these publications solely within yourenterprise provided that all proprietary notices are preserved. You may not makederivative works of these publications, or reproduce, distribute or display thesepublications or any portion thereof outside your enterprise, without the expressconsent of IBM.

Rights

Except as expressly granted in this permission, no other permissions, licenses orrights are granted, either express or implied, to the publications or anyinformation, data, software or other intellectual property contained therein.

IBM reserves the right to withdraw the permissions granted herein whenever, in itsdiscretion, the use of the publications is detrimental to its interest or, asdetermined by IBM, the above instructions are not being properly followed.

Notices 67

http://www.ibm.com/legal/us/en/copytrade.shtml

http://www.ibm.com/legal/us/en/copytrade.shtml

You may not download, export or re-export this information except in fullcompliance with all applicable laws and regulations, including all United Statesexport laws and regulations.

IBM MAKES NO GUARANTEE ABOUT THE CONTENT OF THESEPUBLICATIONS. THE PUBLICATIONS ARE PROVIDED "AS-IS" AND WITHOUTWARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDINGBUT NOT LIMITED TO IMPLIED WARRANTIES OF MERCHANTABILITY,NON-INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE.

IBM Online Privacy StatementPrivacy Policy Considerations

IBM Software products, including software as a service solutions, (“SoftwareOfferings”) may use cookies or other technologies to collect product usageinformation, to help improve the end user experience, to tailor interactions withthe end user, or for other purposes. In many cases no personally identifiableinformation is collected by the Software Offerings. Some of our Software Offeringscan help enable you to collect personally identifiable information. If this SoftwareOffering uses cookies to collect personally identifiable information, specificinformation about this offering's use of cookies is set forth below.

Depending upon the configurations deployed, this Software Offering may usesession and persistent cookies that collect each user's user name and password forpurposes of session management, authentication, enhanced user usability, andsingle sign-on configuration. These cookies cannot be disabled.

If the configurations deployed for this Software Offering provide you as customerthe ability to collect personally identifiable information from end users via cookiesand other technologies, you should seek your own legal advice about any lawsapplicable to such data collection, including any requirements for notice andconsent.

For more information about the use of various technologies, including cookies, forthese purposes, see IBM's Privacy Policy at http://www.ibm.com/privacy andIBM's Online Privacy Statement at http://www.ibm.com/privacy/details in thesection entitled “Cookies, Web Beacons and Other Technologies” and the “IBMSoftware Products and Software-as-a-Service Privacy Statement” athttp://www.ibm.com/software/info/product-privacy.

TrademarksIBM, the IBM logo, and ibm.com are trademarks or registered trademarks ofInternational Business Machines Corp., registered in many jurisdictions worldwide.Other product and service names might be trademarks of IBM or other companies.A current list of IBM trademarks is available on the Web at “Copyright andtrademark information” at www.ibm.com/legal/copytrade.shtml.

Adobe, Acrobat, PostScript and all Adobe-based trademarks are either registeredtrademarks or trademarks of Adobe Systems Incorporated in the United States,other countries, or both.


http://www.ibm.com/privacy

http://www.ibm.com/privacy/details

http://www.ibm.com/software/info/product-privacy

Java and all Java-based trademarks and logos are trademarks or registeredtrademarks of Oracle and/or its affiliates.

Linux is a trademark of Linus Torvalds in the United States, other countries, orboth.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks ofMicrosoft Corporation in the United States, other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and othercountries.

Other product and service names might be trademarks of IBM or other companies.

Notices 69

IBM®

Product Number:

Printed in USA

IBM Operations Analytics - Log Analysis: User's Guide · 2 IBM Operations Analytics - Log Analysis:...

Documents

Transcript of IBM Operations Analytics - Log Analysis: User's Guide · 2 IBM Operations Analytics - Log Analysis:...