a quick start guide

Building Search Based Applications

Adam Lotrowski

Magnify Search

1

Agenda

Magnify Introduction

Search Interface Overview

Transforming Raw Data into Search Content

Magnify Indexing Maintenance

Organizing Magnify Search Libraries

Additional Learning Material

Getting Magnify Help

2

Magnify Introduction

3

Convergence of Search and Business Intelligence

4

Single Access Point to all Applications and Databases

5

Single Point of Information Access

IT

BusinessAnalysts

Anyone

Multiple Application Interfaces

Mainframe Servers Reports Documents Images Multi MediaFormsDatabases

Requirements for Successful Implementation

6

BI

Security

GUI

Metadata

IntegrationConnect to 300 data sources; capture document changes in real time; merge documents; and more…

Use metadata to classify documents, to increase the find-ability, and to render dynamically search filters…

Simple search box, powerful guided navigation, interactive tabular view, multi-drill to additional information…

World class Business Intelligence engine that allows you to ask and answer questions additional questions of the searchable content…

Robust security for any data source, including row and column level security for structured sources; integration with other enterprise level security systems…

How Does Magnify Work?

7

The Magnify Process

• Capture content in real time as information is added or edited.

• Catalog content with a search engine index using tags to categorize, secure, rank and present results stored.

• Cost effective and highly scalable solution leveraging open source Lucene.

• Business intelligence reports presenting various ways to view found content.

• Clustered indexes keep search within one application as more information becomes available.

32

1

Search Request

Magnify Results

Report Request

WF Report

Enterprise Content

Transform Content into Searchable Information

Leverage the Power of Search & Business Intelligence

8

SearchKeyword

Query

BINumbers

Query

Information Inquiries

(1) Search for customer account

(2) Search for vendor transaction

(3) Get account balances (4) Get all transactions

exceeding $ X.xx

• Leverage the strengths of each technology• Leverage hardware and software• Simplify development, maintenance & deployment• Offer better and more complete user experience

Search Interface Overview

9

Reference: Magnify’s End User Manual: http://documentation.informationbuilders.com/masterindex/html/html_wf_7703/wf77maggs/wf77maggs.pdf

The Magnify Home Page

10

Magnify Search Results

11

What is a search result?

12

2

3 4

5

1

Part Description #

Title Every search result has a title which is some text that uniquely identifies the item. This can be a combination of one or more fields.

2

Attribute Groups Search results can designate specific attributes for categorization. This is usually high-level groups when aggregated. In the instance of demographics; this would include gender, nationality, and age range.

1

Search Body When users search specific terms, the terms are matched against a master index generated from result’s searchable content. This can include any structured or unstructured database field or document.

4

Unique ID Drill down into more information passing specific values about the search result to the called link in order to filter the next URL by the value selected.

2,5

Other Details Append additional information to a search result in order to display an image, date, etc. 3

Analytics with BI Search: Example

Search for an inventory item in large repositories across many systems

Narrow down the matchesby using suggestions fromthe faceted navigation

Click to convert the resultsto an Active Report toperform further analysis

Analytics with BI Search: Example

You can now pivot the results to find the item that has the highest cost

Then you can click on the item to generate any typeof report

Or you can switch back tosearch results and runany report on any item

Analytics with BI Search: Example

Drill down to any report with relevant informationfor the item of interest

Inventory, sales, and other reports

Guided ad hoc and predictive analytics

Detailed product documents

Magnify Search Tips

Exact Match: Use double quotation marks to search for two or more keywords that must appear together: "Magnify CentSales"

Inclusion: Use a + (plus sign) in front of keywords that must appear in the search results. To search for results that must contain "plasma" and may contain "TV" use the query: TV +plasma

Exclusion: Use a - (minus sign) in front of keywords you want to exclude from your search. To search for results that contain "stores" but not "united states" use the query: stores –"United States"

Boolean: Use logical operators such as OR, AND, and NOT (case sensitive) to link groups of terms together for a more powerful search. To search for results that contain "music teachers" or both "MP3" and "VHS", but not "New York" use the query: ("music teachers" OR (MP3 AND VHS)) NOT "New York"

Wildcard: Use a * (asterisk sign) or ? (question mark) anywhere in a keyword to expand your search. To perform a single character wildcard search use the "?" symbol. To perform a multiple character wildcard search use the "*" symbol. The single character wildcard search looks for terms that match that with the single character replaced. Multiple character wildcard searches looks for 0 or more characters: D?D or D*D

16

Transform Raw Data Into Search Content

17

Reference: Magnify Developer’s Guide: http://documentation.informationbuilders.com/masterindex/html/html_wf_7703/wf77magdev/wf77magdev.pdf

Magnify: Quick Feed

Challenge

Building Search Based Applications (SBA) required specialized skills

Not many people have this skill

Realization: Building Search Based Applications is no different than building WebFOCUS reports

Magnify category fields equal BY fields

Searchable content equal PRINT fields

Search result links equal DEFINE fields with Drill Downs

New Feature: FORMAT MAGNIFY for quick feeds

Create a report in Developer Studio

Change format to Magnify ….. AND YOU ARE DONE!

18

Magnify: Getting Started

Steps to setup search

Review the data and build context BY JOINing common relationships

Identify high-level groups and designate searchable data

Enhance and enrich the data with drill downs and security

Aggregate data BY merging multiple rows into a single search result

Configure the feed and index with Magnify

Schedule feeds for full search life cycle

Immediate Benefits

Discover information with categorizations

Search across tables and rows

Associate search content with BI reporting

19

Course Example: Setup

Open Developer Studio

Create project for ibisamp

OR

Navigate to Data Servers > EDASERVE > Applications > ibisamp

Create Procedure to Review Content

Create a procedure named courses_all in Procedure Viewer

In Procedure Viewer, add a Report object and select the course.mas file

In Report Painter double click on CRSELIST segment to add all fields to the report canvas (or double-click on each field individually)

Save and Run

Close procedure

20

Review All The Data

21

Identify Categories and Searchable Content

22

Categories

Identify Categories and Searchable Content

23

Categories

Identify Categories and Searchable Content

24

Searchable Content

Course Example: Generate Search Result Metadata

25

Create a Procedure to Transform the Content – Part 1

Create new procedure named courses_transform in Procedure Viewer

In Procedure Viewer, add a DEFINE object selecting the course.mas file

Add the Defined fields:Field Format Value

SYSSOURCE A25 'School Courses'

TUITION_RANGE A50 IF COURSE.CRSELIST.TUITION LT 1000 THEN '1. LESS THAN 1K'ELSE IF COURSE.CRSELIST.TUITION LT 2000 THEN '2. 1K-2K'ELSE '3. GREAT THAN 2K'

SEARCHTITLE A2000 '<b>Course Name: ' | COURSE.CRSELIST.CTITLE | ' (' | COURSE.CRSELIST.COURSECODE || ')</b>'

TITLE_URL A2000 '/ibi_apps/WFServlet?' || 'FXK=' || COURSE.CRSELIST.COURSECODE || '&' || 'IBIF_ex=courses_detail' || '&' || 'IBIAPP_app=ibisamp'

LINK_DISPLAY_NAME1 A2000 'Search Other Department Courses'

LINK_URL1 A2000 '/ibi_apps/search?' || 'q=' || COURSE.CRSELIST.SOURCE

LINK_DISPLAY_NAME2 A2000 'All Courses in this Subject'

LINK_URL2 A2000 '/ibi_apps/WFServlet?' || 'FXK=' || COURSE.CRSELIST.CLASSIF || '&' || 'IBIF_ex=courses_similar' || '&' || 'IBIAPP_app=ibisamp'

HARDCODED A50 ' schools course '

Course Example: Define Transformations

26

Create a Procedure to Transform the Content – Part 2

Create a Report Object, selecting the course.mas file

Add the Fields and Set the Column Title as:Field Column Title (Case Matters)

SEARCHTITLE SearchTitle

COURSECODE MagnifyID

COURSECODE WF_INDEX_UNIQUE_KEY

SYSSOURCE C1_Source_System

SOURCE C2_Department

CLASSIF C3_Subject

TUITION_RANGE C4_Tuition_Range

TITLE_URL

LINK_DISPLAY_NAME1

LINK_URL1

LINK_DISPLAY_NAME2

LINK_URL2

Field Column Title (Case Matters)

DESCRIPTN1 S_DESCRIPTN1

DESCRIPTN2 S_DESCRIPTN2

DESCRIPTN3 S_DESCRIPTN3

CTITLE S_CTITLE

COURSECODE S_COURSECODE

SYSSOURCE S_SYSSOURCE

SOURCE S_SOURCE

CLASSIF S_CLASSIF

HARDCODED S_HARDCODED

Required: SearchTitle, MagnifyID, and WF_INDEX_UNIQUE_KEY Always Recommended: Categories, Magnify Meta Tags, and Searchable Content Save, Run an the Close

Enhance And Enrich The Content

27

Enhanced Title Unique IDs Categorizations Title Drill Down Link

Enhance And Enrich The Content

28

Additional Drill Down Links Searchable Content

Course Example: Connect to Magnify

29

Create a FORMAT Magnify Procedure – Part 1

Copy and paste the procedure: courses_transform.fex

Rename the new procedure to: courses_magnify.fex

Right-click on courses_magnify.fex select Edit in Associated Tool

Before the DEFINE object, add the below ENGINE objects:

Engine Connection SET parameters

MAGNIFY MY_PC 'http://localhost:8080/ibi_apps/xmlfeed'

MAGNIFY -- no connection -- BASEURL=http://localhost:8080/ibi_apps/WFServlet

MAGNIFY -- no connection -- DATASOURCE=school_courses

MAGNIFY -- no connection -- BATCHSIZE=5

Required: Connection Setting, BASEURL, DATASOURCE

Always Recommended: BATCHSIZE

Course Example: FORMAT MAGNIFY

30

Create a FORMAT Magnify Procedure – Part 2

After the ENGINE objects, add the SET object

Used Settings: ASNAMES

Current Value: MIXED

Save

Open the Report Object and Change the Format Type

Click Options

Select Output Format

Expand Unstyled Formats

Select Format Magnify (MAGNIFY)

Click Apply, OK

Save, Run, and Close

Run and Confirm No Errors

31

Review Search Based Application

32

Course Example: Drill Downs

33

Create Drill Down Reports for Course

Create Course Detail Report

Copy courses_all, paste and rename to courses_detail

Add WHERE COURSECODE EQ ‘&FXK’;

Change Output to FORMAT AHTML

Create Similar Course Report

Copy courses_detail , paste and rename to courses_similar

Edit WHERE to use field CLASSIF EQ ‘&FXK’;

Course Example: Next Context Library

34

Create New Index Library

Copy courses_magnify.fex, paste and rename to courses_subjects_magnify.fex

Right-click and Edit in Associated Tool

Change ENGINE MAGNIFY SET DATASOURCE=school_subjects

Add ENGINE MAGNIFY SET DELIMITER=#!#

Modify DEFINE Field: SYSSOURCE/A25='School Subjects';

Modify DEFINE Field: SEARCHTITLE/A2000='<b>Subject: ' | CLASSIF | '</b>';

Modify DEFINE Field: TITLE_URL/A2000=

'/ibi_apps/WFServlet?' || 'FXK=' || COURSE.CRSELIST.CLASSIF

|| '&' || 'IBIF_ex=courses_subject' || '&' || 'IBIAPP_app=ibisamp';

Modify DEFINE Field: HARDCODED/A50=‘ schools subject ';

In Report Painter: Add column: CLASSIFF and Change to BY field

In Report Painter: Recreate MagnifyID and WF_INDEX_UNIQUE_KEY using CLASSIF column

In Report Painter: Rename Column Title C2_Department to M2A_Department

In Report Painter: Rename Column Title C4_Tuition_Range to M4A_Tuition_Range

In Report Painter: Rename Columns by changing S_ to SA_: S_DESCRIPTN1, S_DESCRIPTN2, S_DESCRIPTN3, S_CTITLE, S_COURSECODE, and S_SOURCE

Save, Run, Close

Create Drill Down Report for School Classes

Copy courses_all, paste and rename to course_subject

Add WHERE CLASSIF EQ ‘&FXK’;

Change Output to FORMAT AHTML

See Both Search Result Types

35

Magnify Indexing Maintenance

36

Reference: Magnify Security and Administration: http://documentation.informationbuilders.com/masterindex/html/html_wf_7703/wf77magsec/wf77magsec.pdf

The Magnify Console

Open a browser

Navigate to: http://<server>:<port>/<context_root>/search/jsp i.e.: http://localhost:8080/ibi_apps/search/jsp

Under Diagnostics

Notice Log Settings

Trace Incoming Files Sent By FORMAT Magnify

Log Feed Time and Search Events

Click on Index Monitor

Shows Active Index Libraries being processed

Feed statistics

37

The Magnify Console

38

Index Repository

Open Window Explorer to access WebFOCUS install folders

Navigate to: ibi\WebFOCUS77\magnify\lucene_index

This is where index libraries are generated by FORMAT Magnify procedures

The value set to DATASOURCE is used as the index library directory name

To Delete an index library application server needs to be restarted to release locks on directory.

39

Addition Helpful Tools

Application Server Log files

Records Magnify errors, logs and traces

Lucene Luke

Free online tool to peer into index library

Site: http://www.getopt.org/luke/

Download: http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CCQQFjAA&url=http%3A%2F%2Fwww.getopt.org%2Fluke%2Fluke.jnlp&ei=4DuPT_itIsbW0QGO4rSvDw&usg=AFQjCNEzskrB-EBOfd-dp8LqlA05dKtXHg&sig2=suUCYWEjqMVAxeVSveThMA

Use it to navigate to the index repository, select a specific index directory to open and view

See how many records indexed, available terms, etc

40

Organizing Magnify Search

41

Reference: Magnify Security and Administration: http://documentation.informationbuilders.com/masterindex/html/html_wf_7703/wf77magsec/wf77magsec.pdf

The Collections File

Navigate to the Magnify configuration files: ibi\WebFOCUS77\config\magnify

Open in text editor: collections.xml

Analyzer Options: This lists all available analyzers which instruct Magnify how to parse incoming feeds and search so content can be matched to searches

Defaults Section: Assigns analyzer and style sheet used by the Magnify Search Based Application

Note the stylesheet attribute value: en/en_stylesheet

Collections Section: Organizes Index Libraries into specific groups and by default set to search all root indexes

Modify the member value to: school.* to globally search all indexes beginning with school

Indexes Section: Lists all available index libraries

Copy existing line and paste once

Replace line one with name and directory: school_courses

Replace line two with name and directory: school_subjects

42

The Style Sheet File

Navigate to the Magnify configuration files: ibi\WebFOCUS77\config\magnify\en\

Open in text editor: en_stylesheet.xslt

Collection Descriptions

This list is presented in the drop down next to the search box on the Magnify Interface

After Default Collection add: Subject,Courses,

Optional change the Default Collection text

Must end in a comma (,)

43

The Style Sheet File

Collection Values

This list matches the order of the Collection Descriptions and maps to the indexes (name) setup in the Indexes section of the collections file

After default_collection add: school_courses,school_subjects,

Must end in a comma (,)

44

Refresh Magnify’s Search Based Application

Open a browser

Navigate to: http://<server>:<port>/<context_root>/search i.e.: http://localhost:8080/ibi_apps/search

Refresh Magnify configuration

Restart Application Server

OR

Issue Application Server Magnify refresh commands

Append to the URL

proxyreload=1 to refresh changes to the style sheet

clearcache=1 to refresh changes to the collections file i.e.: http://

localhost:8080/ibi_apps/search?proxyreload=1&clearcache=1

45

See Changes Live

46

Magnify Examples

47

Call Center Account Lookup

48

Enterprise Content Library

49

Law Enforcement Analytics

50

Root Cause Analysis and Reporting

51

Geographical Intelligent Search

52

Additional Learning Material

53

Examples Provided with WebFOCUS

Installed with the Reporting Server and Developer Studio (ibi\apps\ibisamp)

Procedure: carmgn.fex

Procedure: moviesmgn.fex

Century Electronics Magnify Search Example

Installed with WebFOCUS 7.7.03 and higher

Navigate to (adjusting for your own WebFOCUS config): http://localhost:8080/ibi_apps/search?collections=collections_century.xml&clearcache=1&proxyreload=1

You can search on terms in the help section. This can also be accessed from any mobile device.

If Developer Studio is not installed, you will need to unzip the ibimagn archive to the ibi\apps directory. This is found in ibi\WebFOCUS\utilities\demos.

Includes all FORMAT Magnify procedures used to create the example, which includes an example for indexing files.

Recommendation: Change the output of each procedure to HTML to see how the data is being transformed into search results and back to MAGNIFY to index the data

Note: To return to default search page, use the URL:

http://localhost:8080/ibi_apps/search?collections=collections.xml&clearcache=1&proxyreload=1

54

Getting Magnify Help

55

Magnify Support

Contact me at: adam_lotrowski@ibi.com

What to Collect

Try to reproduce with carmgn or moviesmgn

Screen shots illustrating experience encountered

FORMAT Magnify procedure (sample data set if needed)

Zip Index Library (ibi\WebFOCUS77\magnify\lucene_index\<name>)

Zip the config\magnify directory

Tomcat Log file: stdout (zip directory)

Traces

Incoming Documents

Log File Traces ** Some items will depend the tasks being performed

56

Thank You!