Immediate Insight User's Guide€¦ · application initialized datastore initialized Do you want...

Immediate InsightUser's Guide

Dec2015

CopyrightCopyright 2015-2016 FireMon, LLC. All rights reserved. This product and related documentation areprotected by copyright and distributed under licensing restricting their use, copying, distribution, anddecompilation. No part of this product or related documentationmay be reproduced in any form or byany means without the written authorization of FireMon, LLC. All right, title, and interest in the productshall remain with FireMon and its licensors.

This product and related documentation are provided under a license agreement containing restric-tions on use and disclosure and are protected by intellectual property laws.

This product and documentationmay provide access to or information on content, products, and ser-vices from third parties. FireMon, LLC is not responsible for and expressly disclaim all warranties ofany kind with respect to third-party content, products, and services. FireMon, LLC will not be respons-ible for any loss, costs, or damages incurred due to your access to or use of third-party content,products, or services.

The information in this document is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing..

FireMon is a registered trademark of FireMon, LLC. All other products or company names mentionedherein are trademarks or registered trademarks of their respective owners.

Immediate Insight User's Guide

Contents

Copyright 2

Contents 3

About the Immediate Insight User's Guide 10

Document Structure 10

Document Typography 10

About Immediate Insight 11

Immediate Insight IT Data Discovery Methodology 11

Chapter 1: Installation 12

System Recommendations 13

Deploy the Virtual Appliance 14

Initial Setup Overview 14

Deploy on VMWare 14

Initial Configuration 15

Initial Setup 15

Configuration Notes 15

Install on VMWareWorkstation 17

Step 1: Download Immediate Insight 17

Step 2: Import the OVA into VMWare: 17

Step 3: Adjust the settings 17

Step 4: Log on from the VMWare Console 17

Step 5: Log on from a browser 18

Step 6: Connect data sources 18

Installation - Non-clustered 19

Step 1: Download Immediate Insight OVA and License 19

Step 2: Import the OVA into VMWare 19

Step 3: Adjust the VM settings 19

Installation - Clustered 20

Sample Appliance Setup Session 22

Installation - Agent Only 25

Contents: Contents 3

Before Beginning Agent Installation 25

Installing Agent Only on a RemoteMachine 25

Log into Immediate Insight Server GUI, in order to use remote agent 26

Sign In to Immediate Insight 27

For Evaluation Users 27

About License Keys 28

Activate a License Key 28

View Current License Limits 28

Sign Out of Immediate Insight 28

Chapter 2: Administration 29

Key Administrative Tasks 30

How Immediate Insight Communicates 31

Open Ports 31

API Access 31

Security 32

Enabling Encryption 32

Certificates 32

About Users 33

User Types 33

Add a User 33

Data Administrator Accounts 34

CLI Admin PasswordManagement 34

Data Repositories 34

Add a New Repository 34

Manage Data Flow 35

Data Collection 35

Getting Data into the System 35

Set Up Data Collectors 35

Standard Collectors 35

To permanently mount a remote server’s network share 36

To temporarily mount a network share (mount will be lost if the appliance reboots) 36


4 Contents: Contents


Add a New Data Collector 36

Additional notes about adding a data collector: 37

Immediate Insight Collector 38

Set Data Retention 39

Managing Data 39

Deleting Old Data 39

Work with Email 40

Inbound Email 40

Email Alerts 40

Setting up Email Alerts 40

Outbound Email 41

SystemManagement 43

CommandReference 43

Configuration and Diagnostic Files 46

Config Files 46

Diag Files 46

About the Data Router 47

The Data Router: Processing Data as it Arrives 47

Quick Routes 47

Create a Data Route 48

Custom Feeds 50

Create a Custom Feed for a Data Route 50

Edit a Data Route 51

Delete a Data Route 51

Manage Routes 52

Remote Agents 53

Example of Commands using Streaming Format Structure 53

Examples of Commands using Request-Response Structure 54

NMAP 54

Iperf Bandwidth Check 54

OpenVAS 54


Netstat 54

MTR 54

Traceroute 54

Request-response format structure for workflow-initiated commands 54

Example of Command using Request-Response Format Structure forWorkflows 55

NMAP 55

Variables 55

Command Invocation Options 56

Output Options 57

Workflow Tracking 58

How it works 58

Overview 59

Workflow Linking and Interaction 59

Equipment Orders Workflow 59

PerformanceMonitorWorkflow 60

Use aWorkflow 64

Add aWorkflow 64

Edit aWorkflow 67

Delete aWorkflow 67

Delete a State or Transition 67

Workflow Troubleshooting 68

Data Route Actions 68

Custom Action Scripts 70

Loading Custom Scripts 70

Writing Custom Scripts 70

Utility Functions for Custom Actions 71

Parsing 71

Example: 71

Chapter 3: Usage 73

The HomePage 74

Return to the HomePage 74




Search from the Home5

Search Box 75

Time Selector 75

About Settings 76

Change Password 76

Activate License 76

About the DataFlow Page 77

Import 77

Drag & Drop Import 77

Scratch Pad 78

Import Settings 78

About the Firehose Page 79

About the Search Term Box 79

Apply Filter 79

Pause 79

Perform a search 79

Activity Chart 80

Activity Chart Options 80

Details 80

Live Event Count 80

Event Data 80

About the Search Page 81

Click Navigation 82

Search Page Sections 82

Details 83

Search Panels 84

Details 85

OneClick Analytics: Action 85

Add Note 85

Spam 85

Action 86


Auto-tag 86

Perform a Search 86

Search Terms 87

Key Concepts 87

Qualifiers 87

Grouping 88

QuotationMarks 88

Wildcards 88

Metadata 89

Time 89

About theMicroscope Page 90

About the Reputation Page 91

Add Reputation Information 92

Add Bulk Reputation Information 92

Delete Reputation Entries 93

About Pinboard and Bookmarks 94

Pinboard 94

Bookmarks 94

Add a Pin or Bookmark 95

Manage Pinboard and Bookmarks 95

Share a Session 96

Quit Sharing 96

Shareboard 96

Chapter 4: API 97

APIs: Getting Data Out 98

HTTPS Query API 98

REST API Examples 99

API Credentials 100

REST API Certificates 101

Certificates 101

Command-Line Client 102




Installation 102

Command-Line Options 103

Command-Line Examples 103



About the Immediate Insight User's GuideThe purpose of this guide is to help you effectively use the Immediate Insight application. In this guide, youwill find descriptions of the features and explanations of common processes youmight perform in the applic-ation.

Document StructureThis guide contains the following chapters:

l Chapter 1: Installation—details how to deploy Immediate Insight to a virtual machine environment

l Chapter 2: Administration—details how to perform a variety of administrative tasks, such as set-ting up users and collecting data into Immediate Insight. Performing administrative related tasks willrequire a user with administrative privileges.

l Chapter 3: Usage—details how to search and analyze data in Immediate Insight, and get themostout of the application. User privileges also allows for ad-hoc import of data when needed.

l Chapter 4: API— is for system integrators who wish to create integrations between ImmediateInsight and third-party systems.

Document TypographyButtons and graphical user interface (GUI) commands appear in bold. Example: Click Save.

Commands are preceded by the verbs "click" and "right-click". These instructions assume default mousesettings on standard PC mouse devices, where the left mouse button is used for primary navigation ("click")and the right-mouse button is used for additional options and context menus ("right-click").

Code and command prompts appear in Courier New text.

Variables appear in Courier New italics. In many cases, data that you enter or data that the systemreturns is specific to your system. For example, when you ping the Application Server, you will enter the IPaddress of your Application Server, not the sample data shown. Example: ping 192.125.27.25

Additional instructions may appear inside parentheses. Example: Host/IP: (for your Application Server).

A Notewill appear in a green box to provide special information about a feature or process. A notemayalso stress the importance of the information that is provided.

Example code or example output will usually appear in a blue box.

A Caution!will appear to alert you to potential data corruptions that could occur if you proceed with anaction.

A Prerequisitewill appear to alert you to additional steps that must be taken before continuing with a pro-cedure.

: About the Immediate Insight User's Guide 10


About Immediate InsightFireMon Immediate Insight collects and correlates all IT data to help analysts and operations staff increasevisibility into the data and reduce the time and effort spent on correcting issues.

Immediate Insight brings Google-like simplicity and speed to data analysis and discovery. It mergesmachine learning, correlation and natural language in a simple, workflow-centric interface to reveal rela-tionships in the data that users didn’t even know to look for. It transforms organizations from a ‘data as lastresort’ mindset to the ‘data first’ practice necessary to enhance security, performance and operations.

Immediate Insight’s real-time analysis across data silos provides the timely and detailed operational visibilitynecessary to:

l Identify and investigate the suspicious

l Search for indicators of breach and operational inefficiencies

l Get real-time analysis of security data

l Accelerate incident resolution and reduce escalations

l Automatically connect and correlate data silos

l Stage data for analysis by escalation teams

Immediate Insight IT Data Discovery Methodology

: About Immediate Insight 11

Chapter 1: Installation

System Recommendations 13

Deploy the Virtual Appliance 14

Initial Configuration 15

Install on VMWare Workstation 17

Installation - Non-clustered 19

Installation - Clustered 20

Sample Appliance Setup Session 22

Installation - Agent Only 25

Sign In to Immediate Insight 27

About License Keys 28

Sign Out of Immediate Insight 28


System RecommendationsReal-time search and analytics is a resource-intensive process and requires a fair amount of storage for index-ing and workspace. Select a storage system with decent write performance (IOPS). Since this type of datatypically has a short lifespan and you likely have other copies of it, high-availability storagemay not be animportant concern.

Storage requirements vary with data type – data is compressed on disk, but metadata gets added and auto-matic indexing requires working space. A reasonable rule of thumb is 1 GB of storage per 1million activerecords. 500GB is typically a good recommended size. Once storage is provisioned, extending it afterwardis challenging, so it is preferable to provide as much disk space as possible at initial installation.

The distributed architecture of Immediate Insight allows you to scale up by adding additional nodes and clus-tering them together, or by increasing physical resources of this single node.

The following are theminimum recommended system requirements for Immediate Insight.

RAM andmemory recommendations:

l Recommend 32GB RAM, 8 virtual CPUs for production

l Minimum 16GB RAM, 4 virtual CPUs for demonstration

l 16GB is typical for 250million records

l 24GB is ideal for 500million active records

l 32GB is ideal for 1 billion active records

Non-clustered installations:

l 500GB or higher disk space

l 32GB of RAM

l 8 virtual CPUs

Clustered installations havemore robust hardware requirements than standalone installs, it is assumed eachmembermeets the followingminimum specifications.

l 16 or more cores

l 32GB - 64GB RAM, dedicated 1GB+ NIC (bridged if VM)

l Enterprise quality RAID of SSDs or 7200+ RPMmechanical drives for the data. Raid 10 is optional. 1TB of disk space. Use of single drives or consumer grade drives is not recommended. Assigningmul-tiple members to the sameRAID is not recommended.


13 Chapter 1: Installation: System Recommendations


Deploy the Virtual ApplianceInitial Setup OverviewThe following is an overview of the initial installation process.

1. Download the Immediate Insight file.

l It is packaged as anOVA file for VMWare ESXi5 or later.

Note: VMWareWorkstation 8+ or VMWare Fusion Version 6+ are also supported, but these areprimarily recommended for demonstration or testing purposes. See the steps for installing a Imme-diate Insight using VMWorkstation.

2. Deploy the file in the hypervisor, assign data storage to it, and launch.

3. Log on to the Immediate Insight console, assign an IP address, initialize the data storage, and startthe application.

4. Log on from a browser and connect data sources.

Note: Chrome is the supported browser for Immediate Insight.

Deploy on VMWare1. Download the Immediate Insight OVA file provided by FireMon to your computer. The file is approx-

imately 1 GB.

2. From the vSphereManagement Client, select File > Deploy OVF Template... to import the down-loaded file.

3. Click Edit Settings to adjust thememory and storage, and to assign data storage.

4. Adjust thememory and processors as needed. 32GB of RAM and 8 virtual cores is typically a goodsize.

5. Click theResources tab, and then select theReserve all guest memory (All locked) check box.This will provide optimal application performance.

5. Click Add and follow the wizard to add a second virtual hard disk to hold your data. Use of SSD, orfast mechanical drives of 7200 RPM+ are strongly recommended for good performance. 500GB ormore is typically a good size.

Note: If you need to save space it is permissible to thin provision the drive, however thick pro-visioning provides the best performance and is recommended. Store the Virtual Disk as a single file.

6. Save changes, restart the VM and open a console to it.

Chapter 1: Installation: Deploy the Virtual Appliance 14

Initial ConfigurationThese instructions assume you are running the VM in your own hosted environment, for cloud install-ations, please contact FireMon Immediate Insight Support.

Once the application is running in the hypervisor, log on from a console to set an IP address and initialize stor-age.

Console Administrative User: Insight

Default Password:WhatsHappeningNow

The appliance is basically just a stock installation of Ubuntu Server 14.04 LTS, so it has good connectivityand driver support, especially for virtual and cloud environments. The Immediate Insight software runs inuser space and has very few dependencies, so you are free tomake changes to the Linux environment asneeded to integrate with your environment. Standard Linux utilities such as apt-get and editors such as nano,vi, and joe are available.

Note: Run all tasks as the Console Administrative User (CAU). If you need superuser privileges, prefacethe commandwith sudo. Do not attempt to makes changes as user "root" as this can cause conflictswithin the system.

Caution!Do not attempt the change the default password for the console administrative user until theinstallation process is fully complete.

Initial SetupInitial setup is done with the install command.

To verify that installation was successful, use the status command. It should show all 5 system processesas “running” and Data Storage should show 1% used.

If you see “not running” or no number appears for pct used then theremay be a problem. First try using thestart command to resolve the issue, if this doesn’t work, contact FireMon Immediate Insight Support forassistance.

Note: After full reboot of the VM, if Immediate Insight processes don’t start automatically, youmust typestart from the console or SSH session.

At this point you can log on from a browser. You will only need the CLI if you want to add users, mount sharesfor data access, or do other privileged system administration tasks.

Configuration Notesl IP Address: If youmake amistake on IP address, you can use the set-ip command to change it later. You can type “dhcp” in place of IP address for automatic address assignment You can also edit the/etc/network/interfaces file directly if you need to usemore complex configuration options such asIPv6.


15 Chapter 1: Installation: Initial Configuration


l Time: Time is typically acquired from the network automatically, but you can use the date commandto verify. set-timewill force the system to try and update its time via NTP. You can use the set-timezone command to change timezone.

l Update: The software update is done as part of the initial install command, but if you wish to updatein the future, just type update. The version commandwill tell you what version you are currently run-ning.

l Password: To change the system administrator (CLI) password, use the set-password command.This password is also the sudo password for privileged commands.

l SSD: Default assumes amechanical hard drive, but if you are using a SSD you can change the set-ting to optimize Immediate Insight to make use of more efficient IO capabilities of SSD. To enableSSD, enter the command: set-ssd enable

l SSL: Default access is by HTTP. To change to HTTPS, use the command: set-ssl on and thenapply the change by using the command: reload server.

The system uses self-signed certificates by default. You can install your own certificates byplacing them in the app/config/certificates directory.

Note that although the Immediate Insight client runs in the browser, it is a web applicationrather than a web page, so the browser enforces greater security requirements. After log on,the client opens two separate websocket channels to the server, a control channel (port 3201)and a data channel (port 3801). If you are using self-signed certificates some browsers, suchas Firefox, may require you to authorize access for each channel separately.

Note: Chrome is the supported browser for Immediate Insight.

Chapter 1: Installation: Configuration Notes 16

Install on VMWare WorkstationTo install a non-clustered version of Immediate Insight for demo purposes, complete the following steps.

Step 1: Download Immediate Insight1. Download the Immediate Insight OVA file provided by FireMon to your computer. The file is approx-

imately 1 GB.

Step 2: Import the OVA into VMWare:1. Click File > Open a Virtual Machine.

2. Open theOVA file you just downloaded.

Step 3: Adjust the settings1. Add a second virtual hard drive for data storage. We recommend 200 to 500GB of disk space.

Note: Immediate Insight is write intensive. Choose a datastore with good performance.

2. Store the virtual disk as a single file.

3. Click Don’t Allocate All the Disk Space Now.

4. Set the RAM andCPUs. FireMon recommends 32GB of RAM and 8 virtual CPUs.

Note: Theminimum required is 16GB of RAM and 4 virtual CPUs. Reserved RAM is also recom-mended.

5. In Network Adapter, select Bridged.

6. Start the VM.

Step 4: Log on from the VMWare Console1. Connect to the virtual machine through the console in the VMWareWorkstation.

2. Log on to the Immediate Insight CLI. Enter the user name: insight and the password:What-sHappeningNow.

3. Type install and follow the prompts to configure your system.

4. The virtual machine prompts you to install a server or agent. Select server.

5. If you have a SSD drive, typeY and press Enterwhen prompted.

6. Enter the IP address. If it is dynamic, type dhcp.

The virtual machine asks if you want to join a cluster.

7. TypeN and press Enter.

8. Type sudo reboot.

Note: The password for sudo is alsoWhatsHappeningNow.


17 Chapter 1: Installation: Install on VMWareWorkstation


9. Login and type update to verify you have the latest version of code.

10. To launch the software, type start.

Step 5: Log on from a browserChrome is the supported browser for Immediate Insight.

Note: If you don’t know your IP address, type ifconfig in the console to retrieve it.

1. In Chrome, go to http://your-ip-address:3201.

Note: To use HTTPS, type set-ssl at the CLI.

2. Log on as user: admin and password: admin.

3. When you are prompted for a license key, cut and paste in text from the .lic file provided by FireMon.

Step 6: Connect data sourcesFormore information, see the Immediate Insight User's Guide.

1. Go toData Flow > Import.

2. Drag a grouping of LOG or PCAP files under Import as Lines or point log streams at the IP addressof Immediate Insight using port 514 or 3000.

3. Go to theSearchmenu, and click Search.

4. Confirm that data appears.

Chapter 1: Installation: Step 5: Log on from a browser 18

Installation - Non-clusteredInstalling Immediate Insight on VMware ESXi is the preferred, and typical, method in production Enterpriseenvironments.

Note: For testing purposes, it is also possible to install using VMWorkstation.

To install a non-clustered Immediate Insight, complete the following steps.

Step 1: Download Immediate Insight OVA and License1. From the FireMonUser Center, download the Immediate Insight OVA file to your computer. The file is

approximately 1 GB.

2. Download the Immediate Insight license file your computer.

Step 2: Import the OVA into VMWare1. Download and login to VMWare ESXi Console using vSphere Client.

2. Click File > Deploy OVF Template.

3. Specify the OVA file you downloaded from the FireMonUser Center.

4. Select Thin or Thick provision, depending on your available disk space.

Step 3: Adjust the VM settings1. Add a second virtual hard drive for data storage. FireMon recommends 500GB or higher disk space.

2. Create a new virtual disk.

3. Select Thin or Thick provision, depending on your available disk space.

Note: Immediate Insight is write intensive. Choose a datastore with good performance.

4. Store the Virtual Disk as a single file.

5. Set the RAM andCPUs. FireMon recommends 32GB of RAM and 8 virtual CPUs.

Note: Theminimum required is 16GB of RAM and 4 virtual CPUs. Reserved RAM is also recom-mended.

6. Start the virtual machine.


19 Chapter 1: Installation: Installation - Non-clustered


Installation - ClusteredImmediate Insight virtual appliances can optionally be clustered together, allowing for greater capacity, per-formance, and data collection rates. All the cluster member instance’s resources automatically become avail-able to a common event store, which can be searched and analyzed as if they were a single set of data.Once installed and configured, the commonCluster can be searched from any of themember instances.

The following clustered installation instructions make the following assumptions:

l Each Immediate Insight member belonging to a cluster must have a static IP address

l Each Immediate Insight member belonging to a cluster must be on the same version

l Each Immediate Insight member belonging to a cluster must have the same password for the defaultadmin & insight user accounts

l Each Immediate Insight member belonging to a cluster must have the same hostname (for typical vir-tual appliance install this is ‘insight’)

l This is a fresh install (not an upgrade or migration of an existing Immediate Insight system)

l Cluster members are connected with high capacity LAN connections (WAN connections are stronglydiscouraged)

l For best performance, all cluster member servers should be on the same hardware

Note: If your scenario deviates from any of the assumptions above, please contact the FireMonImmediate Insight support team for assistance before proceeding.

For a clustered installation, complete the following steps.

1. Install the first virtual appliance.

Complete the installation and configuration of your first Immediate Insight instance as a nor-mal standalone system. Youmust use a static IP. Make note of the IP address configured,you will need this in step 2.

2. Install the second virtual appliance.

This time, your installation configuration will specify that the instance is part of a cluster. Joining theIP address of the first instance from step 1.

a. EnterY to answer the question: Do you want this server to be part of acluster [N]?

b. Enter the static IP address from Step 1: IP address of any server in thecluster to join:

3. Verify a successful installation

To verify that you installation was successful, use the status command. It should showinformation about the Cluster and its status, all 5 system processes as “running”, and DataStorage should show 1% used.

Chapter 1: Installation: Installation - Clustered 20

4. Add additional Immediate Insight Virtual Appliances to the Cluster (if applicable).

5. Apply licenses.

The first time that you log on to the Immediate Insight web interface you will be prompted for alicense key. Repeat the licensing process on each of the individual member units.

6. Configure the individual cluster members.

Although the cluster shares a common searchable event store, most items for cluster membersstill need to be individually configured. Depending on need the configurations can be common,or specific to each unit. However since you can Search the common event storage from any ofthemember units, you don’t necessarily need to repeat all configuration on eachmember unit –here are a couple of examples:

l Pinboards / Bookmarks – you could copy these to all themembers, but since a Pinboard/ Bookmark configured on any of themembers has access to all the cluster’s data it isn’tnecessary to do so.

l Collectors – you would usually set these up individually on eachmember for added inputrate.

Almost everything in the GUI is configured individually.

At the CLI, most things are configured individually. This includes users, agents, certificates,and all manual parameter changes to the server settings.

Anything touching the event storage is global. That includes events added from theGUI or fromcollectors. Repositories, queries, event delete are global. CLI commands such as reset-data-store, new-repo, and delete-repo are global since they touch the event storage.

CLI commands for managing the cluster are global: start, stop, update, revert, restart, reload,set-ip, install-tools. All other CLI commands are local. Status and diag are cluster awarebecause they report some information about the cluster, but focus on the local node.


21 Chapter 1: Installation: Installation - Clustered


Sample Appliance Setup SessionA setup session should look something like the following example for a single node installation.

Ubuntu 14.04.1 LTS insight tty1

insight login: insightPassword: WhatsHappeningNow

Welcome to the Immediate Insight Management Console

type 'start' to start the server

for a new installation:

'install' to set up the datastore and other con-figurations'date' to check the time. 'set-timezone' to changetimezone if necessary'sudo reboot' for changes to take effect'start' to start the server and then browse tohttp://this-server:3201

use 'set-agent' and 'set-ip' to change settings later'status' checks the current setup

type 'stop' for a clean shutdown before issuing a shutdown commandto the VM

insight@insight:~$ install

New Installation:

Configure this system as an agent or a server [server]?[sudo] password for insight: WhatsHappeningNowIs the data drive an SSD [N}: N (choose Y if using a SSD)

nn

storage volume initialized

IP Address: 192.168.3.103 (or for dynamic addressing type ‘dhcp’)Netmask: 255.255.255.0Gateway: 192.168.3.1DNS Server: 8.8.8.8

Applying new IP Address so we can download the proper updates

Chapter 1: Installation: Sample Appliance Setup Session 22

If you lose connectivity from the IP change, you may need to loginagain

networking initializedapplication initializeddatastore initialized

Do you want this server to be part of a cluster [N]? N

Installation Complete

Type 'sudo reboot' to reboot

This helps assure that the OS fully applies all IP & disk changesAfter the reboot, type 'start' to launch the application

insight@insight:~$ sudo reboot

make sure you have the latest version of code


insight@insight:~? update

You can now login via the Console or via SSH session:


insight@insight:~$ start

starting search - This takes several minutes for large indices......starting appinfo: Forever processing file: data-marshal-server.jsinfo: Forever processing file: data-marshal.jsinfo: Forever processing file: data-marshal.js..

When you use the status command to verify that the installation was successful, it should show all five sys-tem processes as “running” and Data Storage should show approximately 1% used.

insight@insight:~$ status-----------------------------System Status - Quick Check-----------------------------Immediate Insight version: app-2015-0-0


23 Chapter 1: Installation: Sample Appliance Setup Session


Search engine version: search-1.Personality: serverIP Address: 192.168.3.103

Data Marshal: RunningUI Marshal: RunningMarshal Server: RunningAgent: RunningSearch Engine: RunningSearch Memory: 16GB

Data Storage (pct used): 1%System Storage (pct used): 23%System Log Storage (pct used): 1%

Total System RAM: 32GBFree System RAM: 10GBLinux version: Ubuntu 14.04.1 LTS

Note: If you see “not running” or no number appears for pct used, this may indicate an error. Totroubleshoot, type start. If this does not resolve the issue, contact FireMon Immediate Insight Support forassistance.

Chapter 1: Installation: Sample Appliance Setup Session 24

Installation - Agent OnlyThe optional Agent Only installation of Immediate Insight allows a slimmed-down version of ImmediateInsight to be installed on a remote system such as a laptop or server. The purpose of the Agent is to runremote testing commands (for example, iPerf, port scan, or trace route) from a remote system, and to sendthe test results to themain Immediate Insight system tomake available for searches.

Before Beginning Agent Installationl PuTTY to CLI main Immediate Insight Server that will be used tomanage the Agent and receive itsdata.

l cd /app/config

l cat marshal-settings.conf

l Look for the agent secret line and copy the secret word that follows to notepad or something similar

Installing Agent Only on a Remote Machine

Note: A license key is not required for the Agent Only version of Immediate Insight.

1. Deploy Immediate Insight OVA in VMPlayer or VMWorkstation (Open a Virtual Machine)

2. Edit Virtual Machine Settings

l Change VM settings to 1 GB RAM and 2 Cores

l Add another hard drive to the VM. 20GB is sufficient because you aren’t really storingmuchhere

3. Start the VM (Play Virtual Machine).

4. Log on to the Immediate Insight CLI. Enter the user name: insight and the password:What-sHappeningNow.

5. Type install at the prompt.

6. When prompted change from default install of server to agent.

7. When prompted for the IP address use dhcp (or static IP information if applicable).

8. Accept default NTP servers.

9. Specify the IP address of themain Immediate Insight server being used tomanage this agent.

10. Enter the IP Address of themain server.

11. Accept default port 3205

12. Enter the secret word that you copied to notepad previously. It may or may not already be the same forboth systems, if it’s the same you don’t need to do anything. If not, type the correct one.

Note: Because you are in the CLI, the copy and paste function will not work. You'll need to type in thesecret word.


25 Chapter 1: Installation: Installation - Agent Only


13. Specify the name for the remote agent.

14. Type the command sudo reboot to reboot the system.

15. Log back into Agent.

16. Type the start command.

Note: If you receive an error message that the secret word is incorrect, then PuTTY to the agent, thenuse the command set-agent to correct. After correcting, type reload agent.

Log into Immediate Insight Server GUI, in order to use remote agent

Note: You will need an Immediate Insight Server account with admin privileges. Once installed and star-ted, there is no reason to log on to the remote agent CLI or GUI.

1. Go toDataFlow > Remotes.

You should see your additional remote agent listed.

Note: You won’t see it unless the remote agent VM is online, and has IP connectivity to the Imme-diate Insight server, and then Immediate Insight processes are started on the remote agent using thestart command.

2. Click the + icon to add commands.

3. Click the play button to run entered commands.

Results will show in Search on themain Immediate Insight Server GUI.

Chapter 1: Installation: Log into Immediate Insight Server GUI,in order to use remote agent 26

Sign In to Immediate InsightTo log in to Immediate Insight, complete the following steps.

1. Open a browser.

Note: Google Chrome is the supported browser of Immediate Insight.

2. Connect to http://IP-address-of-server:3201

3. On the Immediate Insight Sign In page, enter the following information:

l User: admin

l Password: admin

Note: You can change this password from the Settings menu after you log in.

3. Click Sign In.

For Evaluation UsersAfter logging in, youmay be prompted to enter a license key.

If you are prompted to enter a license key, complete the following step.

l In theEvaluation License Expired dialog box, click Enter License Key, paste the entire contents ofthe text from the .lic file provided by FireMon in theActivation Code field and then click OK.


27 Chapter 1: Installation: Sign In to Immediate Insight


About License KeysNote: Only an admin user can activate a license.

Immediate Insight ships with a free base configuration which allows concurrent storage of 25million eventsfor active search. For this configuration you do not need to apply a license key.

To purchase a higher volume event search license, please contact Immediate Insight Sales, [email protected].

If you have purchased a higher volume license, you will need to:

1. Download the license (file name ending in .lic) from your account in User Center (https://user-center.firemon.com ) to your computer.

2. Open the license in a text editor, such as Notepad.

3. Copy the entire text of the license key file to your clipboard.

Activate a License KeyTo activate a license key, complete the following steps.

1. On the Immediate Insight toolbar, click theSettings icon, and then click Activate License.

2. In theWelcome to Immediate Insight dialog box, enter the activation code in theActivation Codefield that you copied previously.

3. Click OK.

View Current License LimitsTo view the event search volume limits that are currently licensed, complete the following steps.

1. Go toDataFlow >Monitor.

2. In theSearch Engine > Data section, you will see License Limit in theCurrent Records field.

Sign Out of Immediate Insight

To sign out of Immediate Insight, click theSettings icon , and then click Logout.

Chapter 1: Installation: About License Keys 28

Chapter 2: Administration

Key Administrative Tasks 30

How Immediate Insight Communicates 31

Security 32

About Users 33

Data Repositories 34

Manage Data Flow 35

Data Collection 35

Work with Email 40

System Management 43

Configuration and Diagnostic Files 46

About the Data Router 47

Manage Routes 52

Remote Agents 53

Workflow Tracking 58

Custom Action Scripts 70


Key Administrative TasksThere are a few post install administrative tasks required in all Immediate Insight implementations.

l Ensure appropriate TCP/UCP ports are open on the network so that necessary traffic can pass to andfrom Immediate Insight. Reference How Immediate Insight Communicates.

l Add at least one account with user level permissions, assigning a private data repository to that user,so that the user can search their own data separately from other data if desired. Reference AboutUsers.

l Configure Immediate Insight to collect data to be analyzed. Reference Data Collection.

l You should also familiarize yourself with common system administrations commands. Reference Sys-temManagement.

Optional administrative tasks such as Data Routing, Remote Agents, Workflow, and Custom Action Scriptsare also covered in the Administration chapter.


How Immediate Insight CommunicatesOpen PortsBy default, the appliance accepts connections on the following ports:

l Formanagement: SSH traffic on port 22

l For client access: HTTPS traffic on port 3201 and 3801

Additionally, the application has configurable listeners to receive data that default to the following ports:

l Syslog: 514

l Netflow: 2055 (netflow)

l TCP / UDP: 3000

l Data over HTTP: 3001

l Data over HTTPS: 3002

l PCAP stream: 3003

l 3203

API AccessREST API access uses the sameHTTPS and port number settings as the user interface. The API isenabled by default, using an API key of “api-key”. You can change the API key, addmore, or disable theAPI.

Note: For more information about using the API, please refer to the API chapter.


SecurityEnabling EncryptionImmediate Insight streams data to the client by opening two websocket connections to the browser, a controlchannel and a data channel. By default, Immediate Insight is configured for HTTP. To activate encryption(HTTPS) on websockets, complete the following steps.

1. In the CLI, type set-ssl to enable encryption on browser sessions.

2. Type reload server to make changes take effect.

3. Exit the browser and re-login using HTTPS instead of HTTP (https://ip-address-of-server:3201)

Note: Youmay receive a certificate warningmessage, but you will be able to login after ignoring it.Toprevent the warningmessage, refer to the steps to activate a CA certificate.

CertificatesFireMon recommends the best practice use of matching CA certificates installed in your browser to reducethe possibility of man-in-the-middle attacks and provide a smoother user experience.

During installation, a self-signed rootCA pair is generated automatically in app/config/certs.

Note: You can replace this pair with your ownCA by overwriting the rootCA.key and rootCA.pem files,however the self-signed certificate provided is adequate.

To activate the certificate, complete the following steps.

1. In the CLI, type set-certs followed by reload server.

2. Copy the app/config/certs/rootCA.pem file from the Immediate Insight server to your com-puter using an SFTP client.

3. Load the certificate into your browser.

l In the Chrome browser, go toSettings > Show Advanced Settings > HTTPS/SSL > ManageCertificates > Trusted Root Certification Authorities > Import.

l Specify the rootCA.pem file.

4. Restart the browser.

Note: While the system has a reasonable set of security measures in place, the present release isdesigned to run in a secure and trusted environment. If you have a need to expose it directly to the Inter-net, please call FireMon Immediate Insight support to discuss additional hardening procedures.



About UsersAdministrative tasks are divided such that security and systemmanagement tasks are handled from the CLIand datamanagement is handled from the web interface. The admin account for web access is not used forthe same tasks as the insight SSH administrative account.

User TypesIn Immediate Insight, there are three types of users:

l System Administrator—is a CLI account and is only used for administrative tasks such as user man-agement. This account is only available from the CLI and does not work to access the web interface.

l Data Administrator—can perform datamanagement administrative tasks in the web interface. Thisaccount does not have access to the CLI.

l Data User—cannot perform any administrative tasks. Only has access to the web interface.

Additional user accounts can be created and deleted with the add-users tool within the CLI.

Add a UserTo add a user, complete the following steps.

1. At the command prompt type: add-user –n <user name> -p <password>

2. Replacing <user name> with the actual name of the new user.

3. Replacing <password> with the actual password for the user.

If the user exists, the password will be changed. Otherwise, a new user will be created.

Command String Function

-n or –u or –username username

-i or –index <index name> [default = main-stream]

-p or –password password

-g or –group group name [default = user] (user admin for administrator rights)

-r or –repo private/scratch repo [deault = user-shared]

-a or –acl data rights [default = *] (example: +logs, -priv*)

-c or –config location of the users.conf file [default=config/users.conf]

-l or –list list all users

--delete delete a user (example: user –u <username> --delete)

User Account Commands

Chapter 2: Administration: About Users 33

Data Administrator AccountsTo give a user Data Administrator privileges type: add-user -u <user name> –g admin

Replacing <user name> with the actual name of the new user.

CLI Admin Password ManagementThe SSH password for the CLI administrator can be changed by typing passwd

Note: Data Administrator and Data User account passwords are changed from within the web interface inthe Settings option.

Data RepositoriesIncoming data is stored in repositories. All users have access to the default repository:main-stream. Addi-tional repositories can be created or removed using the Immediate Insight SSH account.

Add a New RepositoryTo create a new repos, at the command prompt type: new-repo <repo name>

To add or remove a repository, use one of the following commands in the SSH account:

l new-repo reponame

l delete-repo reponame

Note: Special characters are not allowed as part of the name.

Permission to access the newly created repository must be set for each user who will be working with it. First, get the list of repositories the user has access to (shown in the private repo field):

add-user -list

The default list is main-static,main-stream. Finally, add the new repository to the list shown in the out-put since the user may have access to other private repository.

l add-user -u username -r 'main-static,main-stream,reponame’

If you delete reponame, user permission needs to be adjusted by setting the list with the repo removed:

l add-user -u username -r 'main-static,main-stream'


34 Chapter 2: Administration: Data Administrator Accounts


Manage Data FlowThere are four main systems for working with data-in-motion. Data flow is managed from the DataFlow page.Many of the features within these areas may require an account with administrator privileges.

l Collectorsare used to get data in.

l Data Routing is used for processing data as it arrives.

l Workflow is used for chain analysis of machine and team data.

l Reputation is used for learning from new data.

Data CollectionGetting Data into the SystemThere are twomodes for adding data to the system streaming and interactive.

l Streaming data feeds, such as syslog and log files, aremanaged on theDataFlow > Collectorspage. The system can get streaming data from following log files on a network share and by receivingdata feeds over a network socket. Collectors can only be configured by an administrative level user.

l Interactive input, such drag & drop and cut & paste, is handled from theDataFlow > Import page.Imports can be done by either users with administrative or user level privileges. Hence this manualImport method is documented in the UsageGuide Chapter of this guide.

Set Up Data CollectorsThe default settings for the virtual appliance include several data collectors pre-configured. Data collectorscan be viewed by clickingDataFlow > Collectors.

Standard Collectorsl Socket Listener (TCP / UDP) is already running on syslog (port 514) and port 3000 and listening forstreams on those port numbers. For example, if you point your device syslog destination to point toImmediate Insight’s IP address.

You can also send in data ad hoc from remotemachines using a tool like netcat (on the remotemachine) to send log files over the network to Immediate Insight.

cat logfile | nc ip-of-collector 3000

or

cat logfile > /dev/tcp/address-of-collector/3000

l Netflow Collector is already running on port 2055. It will auto-detect the traffic type and can handleNetflow 5, Netflow 9, Cisco ASA, IPFix, and Bluecoat Packeteer-2 Formats.

l A log file listener is already configured to watch files in the directories /media/logs, /media/logs1/ and/media/logs2. If youmount your logs at those directories or link to them from there, it should begin

Chapter 2: Administration: Manage Data Flow 35

picking up automatically.

To permanentlymount a remote server’s network share

1. Edit Immediate Insight’s /etc/fstab and add a line similar to this:

//servername/sharename /media/logs cifs username=user,passwd=pwd 0 0

Note: You will need to know a valid username and password for the remote server share you aremounting.

2. Use the command sudo mount –a to activate it.

To temporarilymount a network share (mountwill be lost if the appliance reboots)

1. Use the command: sudo mount –t cifs //servername/sharename /media/logs -ouser=username,uid=1000,gid=1000

2. You will be prompted to enter a password.

Note: CIFS support is enabled by default, but since the appliance is just standard Ubuntu 14.04 Linux, youcan use sudo apt-get to install drivers for other file system types and use other mount methods if youprefer.

l If mount directory does not already exist on Immediate Insight, you will need to create it first and allowwrite permissions:

sudomkdir /media/logs

sudo chmod a+rw /media/logs

l CIFS is the default mount type, but NFS is also supported.

Since the appliance is just standard Ubuntu 14.04 Linux, you can use sudo apt-get to install drivers for otherfile system types and use other mount methods if you prefer.

vi cheat sheet for editing system files: i to insert, escwhen finished, :wq to save, :q! to quit w/o save

Add a New Data CollectorIn addition to handling syslog, you can also configure Immediate Insight TCP/UDP Listeners on different portnumbers. Systems such as Firewalls, IDS, Network etc. can often be configured to stream data out to Imme-diate Insight’s IP address on specified port numbers.

To add a new data collector, complete the following steps.

1. Scroll to bottom of DataFlow > Collectors (Inputs) and click theAdd icon +.

2. A menu of collector choices will appear.

3. Select one to open a configuration dialog box (based on your selection) that will allow you to specifyinformation or accept the default settings.


36 Chapter 2: Administration: To permanently mount a remoteserver’s network share


l The collectorName is used as part of the source tag in the index, so ameaningful name canhelp with grouping data in searches.

l Repository allows the selection of private repos. The –stream suffix simply means it acceptsdata from collectors. The list of available repositories is refreshed only when the dialog is firstopened.

l ThePort used for access.

l Rate Limit helps protect the system from explosions if a source becomes misconfigured andemits excessive data. 0 means unlimited. 1000 would enforce a limit of 1000 records persecond for this collector. This can also provide a type of sampling for excessively chatty andrepetitive sources.

l Data TTL is the amount of time data for this collector is saved, 3d is the default – but longertime to live (TTL) is often possible depending on data input rates and system capacity.

l Multline is used in special circumstances only under direction of support (you can leaveblank)

l TheAllow Mirroring check box is used when you wish to allow a copy of the collected datato be retrieved by another Immediate Insight Collector.

4. Click Add.

Note: The Edit Dialog enables you to later customize the settings of the collector

Additional notes about adding a data collector:

l You can also add other types of collectors, for example; File Listeners for other directories, Mail col-lectors, Web collectors.

l As another example of Adding a Collector, if you need to collect Shares other than /media/logs, selectFile Listener.

l The file watcher is filename-aware, so if a log file is automatically rolled over, renamed or replaced bya new file with the same name, the file watcher will start following the new file of the same name.

Chapter 2: Administration: Additional notes about adding a datacollector: 37

l Use /* at the end of the path to watch an entire directory, such as /var/log/* The directory follower willautomatically detect and follow new files as they are added to the directory.

l Wildcards are supported in file names, such as /media/logs/*.log

l The collector name is used as part of the source tag in the index, so ameaningful name can help withgrouping data in searches. It is also used to help specify data routes.

l TheShow Files button shows a sample of the files that this collector will follow, helping to confirmthat the path is correct and active. The collector will actively follow files that have had recent activity(shown in black). It will also passively detect any new or newly re-activated files and then begin act-ively following them.

l Mode: the default is to ‘Follow’ file which is like a Unix tail, new lines are read into Immediate Insightas they are appended to the file. You can also choose the 'Consume’ mode which will read in new filesin their entirely once.

Immediate Insight CollectorThe Immediate Insight Collector is a special type of Collector, used when you want to retrieve a copy of datacollected on another Immediate Insight system (e.g. for a Hot Standby copy of the data).

Example: On remote Immediate Insight system 192.168.1.117 you previously configured one or more it itsCollectors with the ‘Allow Mirroring’ option enabled. Next on your other Immediate Insight below configure atfollows.

1. DataFlow > Collectors

2. Scroll to the bottom of the Collectors Column, click the + icon.

3. Select Immediate Insight Collector.

4. Complete the configuration dialog box.

5. Click Add.

The Immediate Insight Collector will pull a copy of collected data from the allowed collector (s) on theremote Immediate Insight system specified by the Server IP address. Data will be collected into thelocal Immediate Insight Collectors on the specified Server Port.


38 Chapter 2: Administration: Immediate Insight Collector


Set Data RetentionClick on a collector in the collectors column and adjust the settings in its TTL field as needed. 3d (3 days) isthe default. Use h or d to abbreviate hours or days. (24h, 30d, etc.)

Data retention is set on a collector-by collector basis. Click on a collector from the DataFlow > Collectors >Input Column. Set Data TTL to an appropriate value. For example “3d” would set records that come inthrough this collector to be automatically deleted after 3 days (72 hours):

You can see the volume of data currently being stored on theDataFlow > Collectors page in the Real-timeSearch Index box in the Outputs column.

This will add an entry to the Data Routing table that you can subsequently manage or edit from theDataFlow> Data Routingmenu.

Managing DataImmediate Insight is about working with real-time and current data.

Real-time search data tends to decline in value rapidly, there is typically no need to keep it around in thesearch index for very long. To keep the index sizes under control, the system will automatically purge oldrecords. Youmay separately have a logmanagement or data warehouse application for audit and archivingpurposes.

Normally you set a records retention policy that expires data after a few days to keep the volumewithin thelimits of the hardware it is running on. While storage and CPU capacity are important, RAM is usually thelimiting factor for real-time search.

For example, 32 GB of RAM is typically sufficient to handle up to 1 billion active records. You want to set adata retention policy that keeps the total volume of active records under the threshold of the current system.For example with this amount of memory, if the incoming volume of traffic is around 100million records aday, then 7 days is a good expiration policy. If it is closer to 10million records daily, then 60 days is probablya good expiration policy.

Note: In this release, the system still requires somemanual tuning of expiration policies to keep volumeunder control. In the future we expect this process to be increasingly adaptive and automatic.

Deleting Old DataTo bulk delete old records that are already in the system under a different TTL, complete the following steps.

1. On theSearch page, click the time framemenu and select aCustom Time Period.

2. In theCustom Time Selection dialog box, enter a time range for From and To, and then click OK.

3. Click theSearch Resultsmenu and select Delete Result Records.

4. Confirm deletion of the records, click OK.

5. Click Search, then Delete Result Records from the results .

Chapter 2: Administration: Set Data Retention 39

Work with EmailThe system offers two options for email:

l Inbound: listen to email accounts as a source of input data

l Outbound: send email when an alert condition is triggered

Inbound EmailTo poll an email account, such as a notificationmailing list, complete the following steps.

1. Click DataFlow > Collectors.

2. Scroll to the bottom of the Input column and click theAdd icon +.

3. SelectMail Collector.

4. Configure it for your email system. Substitute server information relevant to the POP email accountyou wish to collect emails for.

5. Click Add.

Email Alerts

Setting up Email Alerts

To set up an email alert, complete the following steps.

1. Before you can setup email alert data route actions, the server settings for sending email must be spe-cified in themarshal configuration file. You can edit this file with a text editor, such as vi app/-config/marshal-settings.conf.


40 Chapter 2: Administration: Work with Email


app/config/marshal-settings.conf

2. Add the following lines, using the values for your email environment.

#

# maximum number of email alerts per address per hour

#

mail.maxPerHour = 3

#

# mail alert settings (account to send through)

#

mail.username = "[email protected]"

mail.password = "test"

mail.host = "smtp.gmail.com"

mail.port = 587

mail.subject = "New Alert"

Note: The default maximum setting is 3messages per hour. You can boost this to a largernumber as needed.

3. After saving the changes, type restart at the command prompt to make the changes take effect.

Outbound EmailTo create a new outbound email route, complete the following steps.

1. Click DataFlow > Data Routing.

2. Click theAdd New Route icon +.

3. In theAction box, type the wordmail.

4. Click theEdit Settings button to open amail settings dialog box.

Chapter 2: Administration: Outbound Email 41

5. Complete the fields.

6. Click Send Test Message to verify that the system was able to find a suitable mail server.


42 Chapter 2: Administration: Outbound Email


System ManagementCommand ReferenceFormost installations, start and stopmay be the only command prompts you need after initial deployment,but the system does provide other commands for customization.

The appliance is a stock installation Ubuntu 14.04 LTS Server. You can connect to it with SSH and performstandard Linux tasks to change passwords, mount data, etc.

Immediate Insight adds the following system commands, which you can type at the Linux prompt after con-necting using SSH or the VMWare console.

Note: These commands only work when logged in as the user ‘insight’. They will not work if you changeto 'root' or another user name.

Command Function

start Starts both the search engine and the application

start-search Starts only the search engine areas of the application

start-app Starts only the data collector areas of the application

start-agent Starts only the remote controller area of the application

reload server Restarts the server daemons

reload agent Restarts the agent server

reload actions Reloads the actions from the app/config/actions directory

stop Stops both the search engine and the application

stop-search Stops only the search engine

stop-app Stops only the data collector

stop-agent Stops only the remote controller area of the application

install Complete initial setup of Immediate Insight

set-certs Load in new root CA. The change takes effect after the next restart.

set-memory Auto adjusts the amount of memory allocated to the search index

set-time Will force the system to try updating its time using NTP

set-timezone Change the system timezone

set-updateproxy Used if Immediate Insight must access the internet via a proxy server. Use this com-mand before running ‘update’

Available Commands

Chapter 2: Administration: SystemManagement 43

Command Function

date Change the time and date, if not set automatically

set-ip Sets the IP address, will take effect after reboot

set-ip dhcp Enables DHCP, useful in a configure remote environment

set-capture Activates an interface in promiscuous mode so it is ready for packet capture usefrom the agent

set-dropbox Installs and configures Samba to enable publishing a CIFS share that other hostscanmount and write files to.

reload server Restart the application, equivalent to stop-app + start-app

reset-settings Resets the collector and other setting to factory default

reset-datastore Resets the

add-user Create new users, change passwords for existing user (add-user-h)

ii Searches from the comman prompt

? List of all CLI commands

new-repo Creates a named repository for data collection and organization new-repo <name>

add-tool Utility for installing/configuring/enabling (apt-get) Linus packages for use with agentsadd-tool <tool name>

add-tool update A convenience wrapper for apt-get update && apt-get upgrade which upgrades allpackages on the Linux system

version Display version

update download and install latest version of Immediate Insight code

revert

revert : rollback to second newest version of code on box

revert latest : roll forward from a revert back to the latest version

revert show : list available versions

revert app-09-23-2012 : move to specified version

status Quick summary of the current system state

Available Commands


44 Chapter 2: Administration: CommandReference


Command Function

show

Commands for system health checks

show status: summary of system status

show issues: - list recent critical errors from the log files

show settings: view of any custom settings in the config files. Makes JSON fileshuman readable for better debugging.

firewall

firewall show: display current port blocks

firewall clear: clears port restrictions, allowing

By default, the appliance blocks access to instances of immediate insight not run-ning on the local host. To expand from a single appliance tomultiple appliances youwill need to remove these limits with the firewall clear command, or put more spe-cific firewall rules for your environment.

Available Commands

Chapter 2: Administration: CommandReference 45

Configuration and Diagnostic FilesConfig FilesThe app/config directory holds config files. There is no reason to change these for most installations, but ifyou need to customize behavior, they can be edited with a text editor.

If settings become corrupted, you can restore them to default with the command:

reset-settings -a

This will reset the settings.conf and collectors.conf files; erasing your current collector settings. Tomake thechanges take effect, type

reload

The previous settings files are stored in app/config with a .old extension in case you wish to recover datafrom them or discuss them with support.

l reset-settings –uwill reset the users file, erasing all of your users and setting up the default admin:ad-min account.

l reset-will erase all of the data in the search engine and restore it to default.

Diag FilesA system diagnostics report can be downloaded from DataFlow > Collectors > Ouputs > Diagnostics >Download.

This may help support with troubleshooting if you run into problems.


46 Chapter 2: Administration: Configuration and Diagnostic Files


About the Data RouterThe Data Router: Processing Data as it ArrivesEach time a new event record arrives at the Immediate Insight server, it is immediately processed throughthe data routing table. Much like the packet routing table on a firewall or ACL list on a router, the Data Routerlets you create sophisticated policies for how each event should be handled.

Route Actions can do a variety of things including:

l Alert (using email or in the UI)

l Drop the record (spam)

l Auto-tag or rewrite the record (making it more searchable and readable)

l Learn specific facts from the event (such as mapping a user name or IP address to an email address)

l Decorate (add previously learned facts to this record as metadata)

l Feed the record back out (creating custom data feeds)

Note: See the route action list for amore detailed list. It is also possible to create custom actions for veryspecific behaviors or handling unique sets of data using a simple scripting language.

With data routes you can specify standard handling of incoming records, such as auto-tagging, deleting(spam), and triggering alerts. You can also work with metadata to learn or add additional insight to a newrecord. You can also create custom action scripts that domore complex combinations tasks.

Data routes are found inDataFlow > Data Routing. A summary table is displayed showing the routespresently active.

Routing only happens when a record first arrives. It is not later applied to search results.

Quick RoutesQuick routes provide a way tomatch and group similar records.

The quickest way to setup a new route is from the inlinemenu under an event in the search results.

To set a quick route from the inlinemenu, complete the following steps.

1. On theSearch page, hover over an event to open the inlinemenu.

2. From theActionmenu, select an action item:

l Spam

l Action

Chapter 2: Administration: About the Data Router 47

l Auto-tag

3. AnActions dialog box will open.

l Spam: In theBlock Unwanted Messages dialog box,

a. TheAction option is selected by default.

b. Select aWhat option.

c. Enter a text string tomatch or a comma-separated list of terms.

d. Click OK.

l Actions: In the Take Action on Messages Like This dialog box,

a. Select anAction option.

b. Select aWhat option.


d. Click OK.

l Auto-tag: In theAutomatically Tag Messages Like This dialog box,

a. In theAdd these tags box, enter key words (include the hashtag symbol # before theword).

b. Select a To What option.


d. Click OK.

Note: You can add any additional qualifiers (AND or NOT) or just enter a term tomatch (instead of themore-like-this signature).

Create a Data RouteTo create a new data route, complete the following steps.


2. Click theAdd icon +.

3. Complete theEdit Data Route dialog box. The route editor will let you specify rules for what traffic tomatch and then specify an action to be taken on traffic that does match.

a. Enter aName for the data route.

b. Stage defines the order of execution. Available stages are “arrival” (happens first thing after therecord arrives, nometadata is available), “pre” (after the record has been tagged with sourceinfo, but before other metadata is added), “post” (after standardmetadata has been added), and“final” (last thing, after the record has already been stored in the search index). Leave blank forautomatic selection.


48 Chapter 2: Administration: Create a Data Route


c. Expires defines when the system should auto-delete this filter. Often youmay need an alertor action for only a few days while you work on a specific project. Setting an expiration helpskeep the routing table clean and svelte. You can use shorthand to specify an expiration time,such as “+3d” for three days from now or “sat” for next Saturday.

d. Match is the condition that the new event has tomatch to invoke this route. For example,entering the string “error” would cause this route tomatch all events with the word error appear-ing anywhere in their text.

e. Match Field is the field to match against. If you leave it blank, the default is “text”, which isthe text of the original event. You can view other available fields by clicking onmetadata in theinlinemenu of the search table. Some other interesting fields are “sourceName” and “tag”. “sig” is the fingerprint of the event, enabling you tomatch on similar records.

f. Qualifiers are additional criteria to further narrow down amatch. Similar to how you would con-struct a search query, you can use NOT or AND clauses to refine thematch. Mouse over afield to see help with more details.

g. Action is what to do with the event if it matches this route. See the action table below for avail-able options. Note that (unless a route instructs otherwise), each new event will be checkedagainst all routes and canmatchmore than one. Hit down arrow in this field to see a list of allthe actions presently loaded and available in your system.

l Alert – generate alert within Immediate Insight (or useQuick Route)

l Feed – write search results out to specified file on drivemount (see Custom Feeds)

l FireEye – add additional FireEyemetadata when FireEye logs present

l HL7 – add additional HL7metadata when HL7 logs present

l Mail – send email out for specific search results (seeOutbound Email)

l PaloAltoTraffic – add additional Palo Alto metadata when Palo Alto logs present

l RemoteAgent – call Remote Agent and execute specified Command, for examplePacket Capture (see Remote Agents)

l Spam –mark search result as spam (or useQuick Route)

l Tag – add taggingmetadata to specified searches (or initiate from Search Results)

l Vectra – add additional Vectra metadata when Vectra logs present

4. Click OK.

Note: It is also possible to create custom action scripts.

Chapter 2: Administration: Create a Data Route 49

Custom FeedsThe Data Router lets you re-stream selected records to disk or the network to create your own custom feed. Data routes act on new records immediately as they arrive.

In this example, all records that have been tagged with #SMTP will be written to a file calledmyCus-tomStream.log on the network share that was mounted at /media/logs. You could use the commandautoTagging to tag the different types of records you want to go into the feed. Instead of using theMatchField, another option would be tomatch using only the text of the record to directly select all records in theMatch box, for example any records containing the wordSMTP, leaving theMatch Field box blank. You canhavemore than one route sending data to the same feed.

The feed action takes one property – the name of the file to write to. Use the path where youmounted the net-work share you wish to write to, such as feed(“/media/logs/myFilename.log”) .

Also, streamUDP for Data Routing Action can be used to stream selected data over the network to thirdparty servers or another Immediate Insight server. This feature requires Immediate Insight app-2015-10-07 ornewer.

Create a Custom Feed for a Data RouteTo create a custom feed, complete the following steps.


2. Select an existing route from the list or create a new data route.

3. In theEdit Date Route dialog box, click Edit Settings.

a. Enter aName for the data route.

b. Enter theMatch criteria of events that you want to stream out.


50 Chapter 2: Administration: Custom Feeds


c. Enter anAction, such as streamUDP.

d. Click Edit Settings.

e. Enter the IP address and port of the remote Immediate Insight or other system receiving thedata.

f. Click OK.

g. Click OK.

On the remote server, the selected events appear. For Immediate Insight youmust have a UDP collectorconfigured tomatch the specified port (if you use port 3000 it is there by default).

Note: Only UDP is supported to insureminimal impact on performance. You can collect data as TCP,then stream back out as UDP. Incorrect destinations have little impact.

Note: Events larger than theMTU will be not stream out due to a UDP standard datagram limitation.

Edit a Data RouteTo edit a data route, complete the following steps.

1. Click DataFlow > Data Routing, and select the route to edit from the list.

2. Make the needed changes.

3. Click OK.

Delete a Data RouteTo delete a data route, complete the following steps.

1. Click DataFlow > Data Routing, and select the route to delete from the list.

2. Click Delete.

Chapter 2: Administration: Edit a Data Route 51

Manage RoutesYouwill need to be logged on with an account that has administrator privileges for many routemanagementtasks.

The column on the right hand side of the routing table displays the number of hits. Clicking theRefresh but-ton (immediately above the word hits) will give you an idea if traffic is successfully hitting your route.

In many cases you can generate test traffic to trigger the route simply by typing a few words or pasting a fewlines of data into the Scratchpad.

Using alert(admin) as the action will send all of the resulting hits to the alert feed on the Search page whichwill let you view which traffic is hitting your routing criteria. Once you are satisfied that you are capturing thetraffic you want, you can simply edit the route and replace the alert action with another action.


52 Chapter 2: Administration: Manage Routes


Remote AgentsImmediate Insight can initiate and capture the results of many common network and security analysisproducts. Any program that can be configured to run from the command prompt or shell script can be initiatedby Immediate Insight, either manually or automatically. A copy of the output is streamed in and correlatedwith other infrastructure data.

There are two types of command structures, streaming and request-response. The output for a streamingcommand, tshark for example, is sent to a TCP port collector. A request-response command that outputsdata in a blob, like nmap, is sent to an HTTP collector. There are default remote agent collectors configuredfor TCP port 3001 and HTTP port 3002. If you want to send the TCP or HTTP output to another port, a newTCP or HTTP port will need to be added for the required port(s).

The following are examples of the command structures and commands for streaming and request-responseprograms.

Example of Commands using Streaming Format Structure

Packet Capture: (echo '@@sourceFile:tsharktag'; sudo tshark -b filesize:5000 -b files:5 -w/tmp/tsjunk -t ad -T fields -e frame.number -e col.Time -e col.Source -e col.Destination -ecol.Protocol -e col.Length -e col.Info -E header=n -E separator=, -E quote=d '(host @@a-gentIP)') |nc @@serverIP:3001;sudo rm /tmp/tsjunk*

(echo ‘@@sourceFile:< tag >’; sudo < commandwith any options >) | nc@@serverIP:<TCP

Chapter 2: Administration: Remote Agents 53

collector port >

Examples of Commands using Request-Response Structure

<command with any options> <host> | tee /tmp/<file> | curl –s -XPOST http://@@server-IP:<HTTP collector port >?sourceFile=< tag >--data-binary @- >/dev/null; cat /tmp/<file>; rm/tmp/<file>

NMAP

nmap <address>|tee /tmp/nmapjunk| curl -s -XPOST http://@@server-IP:3002?sourceFile=nmaptag --data-binary @- >/dev/null;cat /tmp/nmapjunk;rm /tm-p/nmapjunk

Iperf Bandwidth Check

iperf -c <address> |tee /tmp/iperfjunk| curl -s -XPOST http://@@server-IP:3002?sourceFile=iperftag --data-binary @- > /dev/null; cat /tmp/iperfjunk;rm /tmp/iperfjunk

OpenVAS

openvas-client -q localhost 9390 admin password /home/insight/scanme.txt /home/in-sight/openvas-output.txt -T text -x; cat /home/insight/openvas-output.txt | curl -s -XPOSThttp://@@serverIP:3002?sourceFile=openvastag --data-binary @-

Netstat

netstat -t <address>|tee /tmp/tracejunk| curl -s -XPOST http://@@server-IP:3002?sourceFile=netstattag --data-binary @- > /dev/null; cat /tmp/tracejunk;rm /tm-p/tracejunk

MTR

mtr --report -c 60 -n <address>|tee /tmp/mtrjunk| curl -s -XPOST http://@@server-IP:3002?sourceFile=mtrtag --data-binary @- >/dev/null;cat /tmp/mtrjunk;rm /tmp/mtrjunk

Traceroute

traceroute google.com |tee /tmp/tracejunk| curl -s -XPOST http://@@server-IP:3002?sourceFile=tracetag --data-binary @- > /dev/null; cat /tmp/tracejunk;rm /tm-p/tracejunk

Request-response format structure for workflow-initiated commands

<command with any options> @@workflowInstanceVal |tee /tmp/nmapjunk | curl -s -XPOST 'http://@@serverIP:<HTTP collector port>?-workflow=@@workflowVal&workflowInstance=@@workflowInstanceVal&sourceFile=req_ls' --data-binary @- >/dev/null;cat /tmp/nmapjunk;rm /tmp/nmapjunk


54 Chapter 2: Administration: Examples of Commands usingRequest-Response Structure


Example of Command using Request-Response Format Structure for Workflows

NMAP

nmap -vv @@workflowInstanceVal |tee /tmp/nmapjunk | curl -s -XPOST 'http://@@server-IP:3002?-workflow=@@workflowVal&workflowInstance=@@workflowInstanceVal&sourceFile=req_ls' --data-binary @- >/dev/null;cat /tmp/nmapjunk;rm /tmp/nmapjunk

Variables

@@serverIP—the IP of the Immediate Insight virtual appliancemanaging this agent.

@@agentIP—the IP of themachine running this agent.

@@workflowInstandVal—theworkflow instance (i.e. IP address).

@@sourceFile—for tagging of command output data

echo’@@sourceFile:<tag>—for streaming, echo themagic header

?sourceFile=<tag>—for request-response, add an argument to URL

@@data_varname—where varname is what appears in event metadata

@@data_text—special case: The entire event is passed in

For example, themetadata value for "source" is passed as @@data_source inthe command prompt. When executed, the shell commandwill see the value"Drag & Drop Uploads (Blobs)" in place of the variable name@@data_source.

Chapter 2: Administration: Example of Command usingRequest-Response Format Structure forWorkflows 55

Command Invocation OptionsStart commandmanually from Immediate Insight UI.

Start command automatically fromWorkflow or Data Router.


56 Chapter 2: Administration: Command Invocation Options


Output OptionsFor request-response programs, output can be sent to the UI by selecting stdin and stderr.

Note: Streaming output to the screen is not supported.

Additional configuration guidelines:

l Interactive commands are not supported and will hang execution when it waits for input. The stop but-ton will recover from this hang condition. All commands must be non-interactive so they can belaunched in an automated fashion.

l All shell commands within a command prompt must respond to the TERM signal to gracefully exit.

l Standard shell script rules apply, and the order of commands determines order of execution within thecommand prompt.

l Only one non-returning command per command prompt is supported.

l Caremust be taken to clean up any generated temp files at the end of the command prompt to avoidrunning out of space.

l Commands requiring sudomust be added to the sudo file.

l Run-time error checking and recovery is the responsibility of the one writing the command. No val-idation or sandboxing is performed at run time.

Chapter 2: Administration: Output Options 57

Workflow TrackingWorkflow tracking correlates team (people) or device (machines) driven business process states with the rawevent data. This enables automatic real-time extraction of actionable information from the raw event streamsthat are captured by II.

How it works

1. Raw events are collected from team andmachine sources.

2. Data of interest is sent to the workflow instances for further analysis. New instances are createdwhen the input is recognized as the first transition in a defined workflow.

3. Workflow instances generate events based on the raw data that are interesting to the organization. These events are collected just like any other events so they can be searched/audited/ acted on/ana-lyzed just like any other events. Workflows themselves are defined by the administrator to model thesteps of business or machine process.


58 Chapter 2: Administration: Workflow Tracking


OverviewThe steps to enable workflow tracking are:

l Define a workflow to be tracked using the workflow editor.

l Add actions to consume the workflow generated events (optional). Email actions can be used toprovide interactive participation within a workflow.

The steps to use output from aworkflow being tracked are:

l Admin communicates specific event keywords that should use when searching for tracked work-flows, alerts that were defined, and workflow generated emails to all II operators.

l Operators should take appropriate action to follow up on any received workflow events, alerts oremail.

Workflow tracking greatly reduces the noise that operators usually face whenmonitoring large amounts ofdata by allowing them to focus on the relevant key events affecting business impacting workflows.

Workflow Linking and InteractionIt is possible to use the output of one workflow as input to another workflow. This advanced technique is use-ful for extracting aggregate or higher level information out of one or more workflows.

In this example, an expense report process is beingmonitored by another process that tracks response time. In addition, themanager can remotely interact with themonitor workflow based on the current status.

EquipmentOrders Workflow

This is the primary workflow process being tracked.

In this workflow, theReview Pending and 2nd Level Review states have times set.

Chapter 2: Administration: Overview 59

In both states, the…active longer than event triggers an alert so the team knows they are running behind.

If the…exited later than time passes, thePerformanceMonitorworkflow is triggered.

Performance MonitorWorkflow

In this workflow, themanager wants to know about slow exit times and take action to resolve the issue.


60 Chapter 2: Administration: PerformanceMonitorWorkflow


The transition for slow review is triggered by the output of theEquipment Orderworkflow.

In the advanced options (accessed by clickingMore after theMatch Field), there is an option to allow work-flow events as input.

Chapter 2: Administration: PerformanceMonitorWorkflow 61

In this case, the event from theEquipment Orders workflow is specified as theMatch value.

To alert a manager of a slow equipment order, themonitor workflow is configured to generate an event whentheSlow_Review_Seen state is entered.

A data route is configured to send email to amanager when theSlow Review Seen state is entered.


62 Chapter 2: Administration: PerformanceMonitorWorkflow


The From field is the reply address account that Immediate Insight is polling for input.

The original event must be included since it contains the unique ID used by themonitor workflow.

To allow themanager to reply back to Immediate Insight, themail data collector has been configured toreceivemail:

The performancemonitor workflow has transitions that are configured tomatch keywords (warning, dis-missal) in themanager’s reply. The instance ID is picked up from the original event that was included in theemail (using@@text).

Chapter 2: Administration: PerformanceMonitorWorkflow 63

Use a WorkflowThemost direct way to use the output of workflow tracking is to search for the events that were generated.

The Instance ID for workflows is always enclosed in the number sign (#).

In this example, the instance FRM:123 was moved to the form_submitted state. Metadata and a tag areadded to the event automatically.

Searching on the tag will return all events related to that particular instance.

Add a WorkflowTo add a new workflow, complete the following steps.

1. Go to theDataFlow >Workflow page.

2. Click theAdd icon .

3. Complete theWorkflow Properties dialog box.

a. Workflow Name cannot be changed after adding.

b. Workflow ID is automatically generated based on the workflow name entered. The ID is usedin generated events.

c. Description of the workflow, this is optional.


64 Chapter 2: Administration: Use aWorkflow


d. Instance TTL specifies the length of time a workflow instance stays active. This is to preventabandoned instances from consuming system resources.

e. Maximum Number of Instances caps the number of instances that can be active at thesame time.

f. You can select or clear check boxes to best meet your needs.

g. Click OK.

Note: All fields, except Description, are required .

4. In theEdit Workflow: <name> dialog box, you will add a number of states that present the processbeingmodeled. This allows the workflow definition to be created in amanner similar to creating aPowerPoint slide. This visual representationmakes it easier to understand the flow and relationshipof the inputs that affect the transitions to different states. In the workflow dialog box, click theAddicon, theAdd State: New State dialog box will open.

a. State Name should be reasonably descriptive.

b. State ID is automatically generated based on the state name entered.

c. Select aState Type.

l Regular—intermediate step in the workflow that leads to another one.

l Stop—the end step in the logical sequence of steps. Transitioning to a stop state alsocauses the workflow instance to finish. More than one stop state is valid. It is recom-mended that all logical paths lead to a stop state.

d. Select or clear check boxes toGenerate these events. One or more events can be selected. The list of available events changes depending on the type and presence of other states.

e. Click OK. Once the state is created, it will appear on theEdit Workflow: <name> dialog box.

5. Repeat step 4 to continue adding states as needed.

Notes: It is best practice to have at least one stop state and provide transitions such that a pathexists from every state to one or more stop states.

6. In theEdit Workflow: <name> dialog box, you will add transitions between states by clicking theTransition icon in theState box and drag to another state box. You will see the color of the state boxchange and theAdd Transition: New Input dialog box will open.

a. Transition Name should be reasonably descriptive.

b. Transition ID is automatically generated based on the transition name entered.

c. Workflow Instance Filter is a regular expression that matches a unique identifier within theincoming event. Be sure the identifier that you choose is contained in all events that will beused as transition inputs for this workflow. This identifier is how the system associates inputsa particular instance of a workflow.

d. Stage specifies when to do the route check. Default is Auto based on action types. Otheroptions are: Arrival, Pre, Context, Past and Final.

Chapter 2: Administration: Add aWorkflow 65

e. Match is a regular expression that matches something in the event that identifies it as a type ofthis kind of input. Make thematch expression as explicitly as possible to avoid detecting thewrong input for this transition.

f. Match Field is the field that Match text will be checked against. Leave blank for the originalevent text. Other options are: Text, Source, SourceFile, and Tags.

g. Click More to access additional parameters for the transition.

l Qualifier Field is the field that Qualifier terms will be checked against.

l AndAny Qualifier is a comma-separated list of additional terms. In addition toMatch,at least one of thesemust alsomatch for the route to be triggered.

l AndAll Qualifier is a comma-separated list of additional terms that all need tomatchfor the route to be triggered.

l Not Qualifier is a comma-separated list of additional terms that all much not match. Ifany match, the route will not be triggered.

h. Click OK.

6. Repeat step 6 to continue adding transitions to connect the states.

7. Click OK.

Here are some helpful tips to make it easier to correctly fill in the Instance andMatch expressions.

l Gather samples of events generated by your devices and/or applications. In some cases, the devicesand/or applications need to be configured to send events or addmore details to events related to theworkflow being tracked.

l Identify the common identifier across all the events that will be associated with a workflow. This canbe a transaction ID, document name, ticket number, etc.

l For each event, determine the kind of input that it is and how it impacts the progress through theworkflow. Add or modify states as needed to cover the steps of the process you want to track.

l There are numerous online resources for regular expression syntax.

l Test your new workflow by importing your sample data using scratchpad or sending it to the TCP col-lector and watching the firehose for your selected generated events.


66 Chapter 2: Administration: Add aWorkflow


Edit a WorkflowTo edit an existing workflow, complete the following steps.

1. On theDataFlow >Workflow page, click theName of a workflow.

2. You can edit the Properties, State or Transition.

l Click theWorkflow Properties icon .

Note: A workflow's Name and Instance ID cannot be changed.

l Click theEdit icon in theState box.

l Click on the Transition line.

3. TheWorkflow Properties orEdit <name> dialog box will open (depending on what you selected instep 2) and you canmake your changes.

Notes: Changing the type of state from regular to stopwill delete all outbound transitions from thatstate because it is not valid to transition from a stop state.

4. Click OK.

Delete a WorkflowTo delete a workflow, click the name of the workflow and on theEdit Workflow: <name> dialog box, clickDelete.

Delete a State or Transition

Caution!Deleting a state will also delete all transitions connected to it.

1. On theDataFlow >Workflow page, click theName of a workflow.

2. Click theEdit icon in theState box or click on the Transition line.

3. In theEdit <name> dialog box, click Delete.

Chapter 2: Administration: Edit aWorkflow 67

Workflow TroubleshootingAlways make sure thematch expression is written as specifically as possible. If it is too general, the inputmay be sent to the wrong workflow.

Always make sure the instance expression accurately matches the unique ID. It is okay that delimiters arepart of thematch.

The file app/logs/data-marshal.log contains workflow tracking details that are useful for troubleshooting.

Avoidmaking changes to a workflow during business hours since changes take effect immediately and couldconfuse the users who are working with instances of the workflow you changed.

Data Route ActionsCreating an action based on a workflow event allows the system to respond to workflow activity.

For example, to send an alert when a form submission is taking longer than it should.

Enable the event of interest in the workflow state: form submitted

Add a data route action to generate an alert when that event is seen. But don’t enter the instance in theMatchField.


68 Chapter 2: Administration: Workflow Troubleshooting


The alert will now appear when the time has elapsed:

Chapter 2: Administration: Data Route Actions 69

Custom Action ScriptsTo createmore sophisticated actions, you can write your own scripts.

Immediate Insight uses a JavaScript scripting environment, which enables quite a bit of power and flexibility.

Keep inmind that if your action will be executedmany times a second, performancematters. Try to keepscripts short and simple. Actions that are only hit once in a while are less of a performance concern.

Loading Custom ScriptsPlace action scripts in the app/config/actions directory. You can upload scripts via SFTP or edit them dir-ectly from an SSH session using one of the command prompt editors (vi, pico, nano, joe). By convention,action scripts should have a .actions extension (example:myCustomAction.actions). You can havemorethan one action script in the same .action file and you can placemore than one .action file in the actions dir-ectory.

Once you have added or changed an .action file, tell the server to reload it by typing@@reload actions intothe search box of the UI:

Alternatively, you can type restart from the SSH command prompt, which will restart the server process.

After reloading the actions, your custom action script will appear as a choice in the Actionmenu, when editingor creating a Data Route.

The following section provides some tips for creating custom actions scripts (for further assistance you canalso contact [email protected] )

Writing Custom ScriptsCode is standard JavaScript. A simple action file looks like this:

//----------------------------------------------------

// Sample Custom Action

//-----------------------------------------------------

var actions = exports;

actions.myScript =

function(data){ var now = new

date() data.myDate = now

return data }

Lines starting with // are comments.

All .actions files must start with the line:


70 Chapter 2: Administration: Custom Action Scripts

mailto:[email protected]


var actions = exports

After that you can have one or more action functions in the file. Action functions look like this:

actions.myScript = function(data){ … your code here … }

In this example, we’ve used the namemyScript for the action. The rest of the line (actions. , function(data))is required and always the same.

Each time the action is called, it receives a data object which contains the text of the data record andwhatever metadata information is available at that time. Typically the interesting values are:

l data.text: the text of the event

l data.source: the name of the collector

l data.sourceFile: the path or port number the collector got this record from

l data._ttl: 3d

At this point in time, the record has not yet been stored and indexed, so you canmodify it and addmetadatato it.

For example, this action would create a new metadata entry called xUppercase for the record that containsthe original message converted to uppercase. You can view themetadata in the details panel of the searchscreen by clicking a panel display icon .

var actions = exports;

actions.myScript = function(data){ data.xUppercase = data.-text.toUpperCase() }

Utility Functions for Custom ActionsImmediate Insight provides a number of utility functions that you can use to work with data.

Parsing

utils.csv(text,delimiter)

l Takes a line of csv data and splits into an array.

l Handles escaping such as quoted fields containing commas or newlines and double-quotes like""foo""

l By default, delimiter is comma or tab, but you can specify other values such as space (" ") or pipe ("|")

Example:

For input data that looks like this: 12345,Jan-24-2013, "Smith, John","Supervisor". Extractthe UserID (12345) and the name and add them to the record as metadata

Chapter 2: Administration: Utility Functions for Custom Actions 71

actions.FindUserID = function(data){

var e = utils.csv(data.text)

if(e.length < 4){ return } //record too short, don’t

proceed data.userID = e[0] || "no id" data.userName = e[2]

|| "no name" data.userTitle = e[3] || "no title" }

The text following the || is the default to use if no value is available from the record

Note: The numbering is zero-based, so the first item is 0 and the third item is 2.


72 Chapter 2: Administration: Example:

Chapter 3: Usage

The Home Page 74

About Settings 76

About the DataFlow Page 77

About the Firehose Page 79

About the Search Page 81

About the Microscope Page 90

About the Reputation Page 91

About Pinboard and Bookmarks 94

Share a Session 96


The Home PageAfter signing in to Immediate Insight, the Home page opens. From this page you navigate to other areas ofImmediate Insight.

Toolbar menu options include access to additional pages. To open, click the page name.

l Search—used to look through past data and expand or narrow scope. Real-time search capabilities.

l Firehose—used to follow data in real-time as it arrives.

l Microscope—used to examine a specific event in greater detail.

l Reputation—used to configure internal reputation. View internal and external reputation.

l DataFlow—used tomanage data flow and connect to new data feeds.

Additionally, from the toolbar you can access bookmarks, help menu and settings.

Icon Description

Click this icon to open a list of available Pinboards. Only available on theHome page or on the pinned search page. Not part of the toolbar.

Click this icon to return to the HomePage.

Click this icon to open Bookmarks. Also used tomanage pinboards and book-marks, show history, share a session or share a configuration.

Click this icon to open the list of HelpMenu items.

Click this icon to open account settings and to log out of the application.

Toolbar Icons

Return to the Home PageYou can return to the home page from anywhere within Immediate Insight. To return to the home page, com-plete the following step.

l Click the Immediate Insight icon in the upper left-hand corner.


Search from the Home PageLocated in themiddle of the page is a search box and time selector.

Search Box

Enter a few words to start the search, For example, a fragment of an error message, a user name otIP address, a file name or page name, a location or data source. You do not need to be specific because youcan easily narrow the results on

l To search, enter the search criteria on the field and press Enter or click Search. The Search pageopens.

Time Selector

Select the time range to search. Immediate Insight is about what's happening right now, so the last hour isusually a good place to start.

l To select a time frame, click the arrow to choose from a list of time-related options.


About SettingsFrom the Settings option, you canmanage your user password, activate a license and log out of the applic-ation. You can also view your logged in as user ID.

Change PasswordAll users canmanage their own passwords.

To change your password, complete the following steps.

1. On the toolbar, click theSettings icon, and then click Change Password.

2. Complete theUser Preferences dialog box.

a. Enter your current password in theOld Password field.

b. Enter your new password in theNew Password field.

c. Enter your new password again in theNew Password Again field.

d. Click OK.

Activate License

Note: Only an admin user can activate a license.

To activate a product license, complete the following steps.

1. On the toolbar, click theSettings icon, and then click Activate License.

2. In theWelcome to Immediate Insight dialog box, enter the activation code in theActivation Codefield, and then click OK.


About the DataFlow PageFor user level access, the DataFlow page is where interactive data can bemanually imported.

For administrator level access, the DataFlow page also allows management of Collectors, Data Routing,systemMonitoring, Remotes, andWorkflow.

There are six options to provide a different perspective of the data.

l Import page can be used for on-demand importing.

l Collectors page displays collector statistics. Themost active collectors are shown at the top. Eachcollector will show separate statistics for each file it is actively following. Drops due do to rate con-trols, if you have applied any, will also be shown here. The display updates every few seconds.

l Data Routing gives you very granular control over the processing of each record as it arrives, muchthe way a network router or firewall gives you granular control over the processing of each packetpassing through it.There are a few one-time configuration options available for data routing that areconfigured on the command prompt.

l Workflow tracking correlates team (people) or device (machines) driven business process stateswith the raw event data. This enables automatic real-time extraction of actionable information fromthe raw event streams that are captured by Immediate Insight.

l Monitor provides an overview of information coming in from various sources.

l Remotespage displays the available remote agents.

ImportInteractive data can bemanually imported from theDataFlow > Importmenu.

Options to select when importing include:

l Tags—you can append tagmetadata tomake an individual import easier to find later

l Repo—you can put your imports in a separate repository frommain stuff streaming in

l TTL—how long the data is kept before discarding (default is 3 days)

l Multiline delimiter to help figure out where one line stops and the other begins (note: this is not nor-mally needed as Immediate Insight does a good job of this automatically, special cases only)

l Import Time—select Original if you want Immediate Insight to try using the time stamp present inthe original data, otherwise select Now to normalize the data to the Immediate Insight clock.

Note: Search is time dependent, if you don’t see data check the time range

Drag & Drop Import

You can select to import targets as lines or as blob. Import As Lines processes each line individually, suchas a log file. Import As Blob processes the entire file as one item, such as with a configuration file. If indoubt as to which type to use, select Import as Lines.

You can import a variety of file types: .rtf, .text (UTF-8, UTF-16), Pcap, Word, PowerPoint, Excel, .pdf, .pst,.zip, .rar, .tar, Gzip, Bzip2, .xz, JSON, and XML.


Note: Nested archives cannot be imported. Only text within files are imported, graphics are ignored.

Scratch Pad

You can enter or copy notes directly into the scratchpad page area of the page that you want to add to asearch. You can copy-and-paste information into theScratchpad, such as lines from an SSH terminal ses-sion or an email.

Import Settings

You can select how imports are handled based on Import Time and Import Type.

For Import Time specify if you wish to use the original source time stamp (where present), or the systemtime stampwhen indexing the records into the system (system time stamp is the default).



About the Firehose PageThe Firehose page is used to view incoming data. It provides a quick real time view to data coming intoImmediate Insight either from the collectors, and/or from drag & drop import function.You can use it for ahigh-level look at your data, but also to confirm that data collection is working.

You can optionally filter the search on simple key works, and pause for easier viewing.

Below is a list of icons that can be found on the Firehose page to help with navigation.

Icon Name Function

Show/Hide Click to show or hide the data stream section.

Show/Hide Click to show or hide the activity graph.

Show/Hide Click to show or hide the details section.

Search Click to search on the term. You will go to the Search page.

Reset Click to erase the current search context. Click again torestore the search.

Firehose Page Icons

The following are sections available on the Firehose page.

About the Search Term BoxThe search term displays only lines matching this term. The text will turn red if the query term entered isinvalid. Searches matchmetadata as well as the displayed event text, so a search term of "syslog" willmatch all records arriving from the syslog server.

Clicking the search button will open the Search page and do a historical search on the term. This is usefulfor terms you recently clicked on in the flow and if you want to look at past details.

Apply Filter

Clicking Filterwill apply a new search term to limit what is displayed. The box will turn orangeif changes to the search term have not yet been applied.

Pause

ClickingPausewill pause the feed so you can read what is there. Data continues to flow aslong as you are on this page, it just isn't displayed. If youmove off of this page, you will needto click Resume to tell the server to restart the feed.

Perform a search

To perform a search, complete the following steps.

1. On the toolbar, click Firehose.

Chapter 3: Usage: About the Firehose

2. In theSearch Query field type your search criteria.

3. Click Filter to limit the search to only the term.

4. Click Pause to visually pause the data stream.

5. Select to includeAll Data orSearch Only.

Activity ChartThe activity chart displays a count of the number of matching events per second. This count is kept by theserver and shows all matches, even if the record was not displayed on the screen due to rate limits.

Activity Chart Options

Click theActivity arrow to change the rate of update for the activity chart.

DetailsClick on the line in the feed in the Details section to see its metadata. This is useful for viewing the name ofthe collection source.

Live Event CountThe live event count contains the line, time and event details of the record. Tomanage bandwidth and per-formance, as well as respect the limits of human visual processing, the live feed will not update faster thanapproximately 12 events per second. If morematches than that are available, the rest will not be sent fromthe server or displayed. A gap in the count indicates this. You can use the Search page if you need to go backand look in more detail at all records in a certain time period.

Event Data

Record data is colorized to highlight possibly interesting terms. Click on a term to set a filterand show only records containing that term. Click on the line number to see details of therecord in the Details section.


80 Chapter 3: Usage: Activity Chart


About the Search PageOn the Search page you can perform a search and then view a variety of search result information, as well asperform most of your detailed analysis.

Navigation is divided into several mains sub-panes. It is helpful to understand these at a high level first.

Below is a list of icons that can be found on the Search page to help with navigation.

Icon Name Function

Panel Display Click to show events panel only.

Panel Display Click to hide top panel.

Panel Display Click to hide details panel.

Panel Display Click to show all panels.

Pinboard Click to open the Add to Pinboard/Bookmark dialog box tosave the query.

Reset Click to erase the current search context. Click again torestore the search.

Entity Quick Access Click to toggle through IP addresses, locations, networks,tags, etc,

Additional Options Click to view: Details, Origin Info, Follow

Change View Options Click to change to: Tools, Time Slices, Subsearch,Details

Quick ChangeOptions Click to quickly move to a new view: Events, Asso-ciations, Clusters, Comparisons, Tags/Notes

Reduce Click to eliminatemost common results.

Colorize Click to highlight known entities or unique terms.

Short Form Click for the short form view.

Flatten Click to reducemulti-linemessages to a single line.

Map View Click to view amap.

Search Page Icons

Chapter 3: Usage: About the Search

Icon Name Function

Zoom Click to zoom in or out of themap view.

Table View Click to view information in table form.

Details View Click to view a list of details.

Scroll Click to scroll through information.

Search Page Icons

Click NavigationImmediate Insight search attempts to highlight things in the data that might be interesting terms and callsthem out with color and underlining. Click a term to pivot the search context to that term. You can useShift+Alt+Ctrl or the Search Action bar in the lower left of the screen tomodify what happens when you clickeach of the buttons.

Click Navigation is available on all pages.

Button Name Function

Drill-down Adds clicked term to the search context to narrow results.

Reduce Clicking will eliminate the clicked term from search context.

Focus Clicking will change the current search context to the clicked term.

Follow Click to view current activity for the clicked term in real-time (Fire-hose).

Subsearch Click to view the clicked term as a subset of the current searchcontext, without changing the search context.

Click Navigation Icons

Search Page SectionsTheEntity Quick Access button will show or hide additional

l Entities - Infrequent Entities

l Sources - Infrequent Sources

l Tags

l Locations - Location Country - Location Region - Infrequent Locations

l Networks - Infrequent Networks


82 Chapter 3: Usage: Click Navigation


l Users - Infrequent Users

l Apps - Infrequent Apps

TheChange View options will open one of four sections.

l Tools offers four menus for quick navigation to the various search views.

l Time Slices shows counts of events. Click the arrow to select other metrics such as peaks, aver-ages, and other variables.

l Subsearch provides a way to easily view the activity of an individual entities within the currentsearch context.

l Details provides metadata for currently selected event. Point to an event to show more options.

TheQuick Change options will open one of five sections.

l Search Results provides access to One Click Analytics.

l Association Analytics summarizes data by counts.

l Event Clusters summarizes data by similarity.

l Activity & Change displays information changes in the data.

l Annotations provides summaries for notes, tags and alerts.

TheAdditional options will open one of three sections.

l Origin Info

l Follow

l Details provides an opportunity to look up reputation data.

Details

To look up reputation data, complete the following steps.

1. Click .

2. Enter search text in theEntity field, and then click Lookup.

To look-upmore detailed reputation data, complete the following steps.

Chapter 3: Usage: Details 83

1. Click .

2. Enter search text in theEntity, Field, Value andExpires fields.

3. Click Lookup.

Search Panels

Clicking one of theChange View options will open one of four panels.

l Tools offers four menus for quick navigation to the various search views.

l Time Slices shows counts of events.Click the arrow to select other metrics such as peaks, aver-ages, and other variables.

l Subsearch provides a way to easily view the activity of an individual entities within the current searchcontext.

l Details provides metadata for currently selected event. Point to an event to show more options.

Clicking one of theQuick Change options will open one of five panels.

l Search Results provides access to One Click Analytics.

l Association Analytics automatically groups search results into Entities, Locations, Addresses,Users, Apps, Networks, and Sources. These are useful for narrowing search results.

o Click the toggle icon to switch from frequent to infrequent results.

o Entities are not only IP addresses. Click

o Click [more] to open a complete list, and then click the arrows to openmore details.

l Event Clusters automatically summarizes and groups current Search Results based on similarity.Event clusters can be helpful when you don't know where to start looking.

o You can switch between Common and Unusual (least common) clusters.

o Click the colorize icon to highlight unique terms.

o Click the reduce icon to eliminate themost common results.

l Activity & Change compares what you are currently searching to an earlier time period (previous, anhour ago, yesterday, last week). This allows you to see if an event results are normal or deviating frombaseline. You can also compare two subsets of the current data to narrow the baseline.


84 Chapter 3: Usage: Search Panels


o You can change the trend inspected by clicking the trend heading (Trending Up, TrendingDown, Active, etc.).

o After selecting the initial time period, you can select another by clicking the currently selectedperiod.

o An event cluster result isn't necessarily a problem, but if it is new or trending up then it mayindicate a need for investigation.

l Annotations provides summaries for any notes, tags and alerts relevant to the current search res-ults.

Clicking one of theAdditional options will open one of three panels.

l Origin Info

l Follow

l Details

Details

To look-up reputation data, complete the following steps.

1. Click .

2. Enter search text in the Entity field, and then click Lookup.

To look-upmore detailed reputation data, complete the following steps.

1. Click .

2. Enter search text in theEntity, Field, Value andExpires fields.

3. Click Lookup.

One Click Analytics: ActionPoint to an event in the Search Results section to access the Actionmenu options.

Add Note

To add a note, complete the following steps.

1. Click Add Note.

2. Type a note with or without hashtags.

3. Click Add Note.

Spam

To block unwantedmessage, complete the following steps.

Chapter 3: Usage: Details 85

1. Click Spam.

2. Complete theBlock Unwanted Messages dialog box.

a. Select anAction to take.

b. SelectWhat to block.

c. Enter matching terms in the box.

d. Click OK.

Action

To take an action on similar messages, complete the following steps.

1. Click Action.

2. Complete theActions dialog box.

a. Select anAction to take.

b. SelectWhat to take the action on.


d. Click OK.

Auto-tag

To automatically tag similar messages, complete the following steps.

1. Click Auto-tag.

2. Complete theAutoTag dialog box.

a. Enter hashtags in the box.

b. Select To What to add the tags to.


d. Click OK.

Perform a SearchTo perform a search, complete the following steps.

1. On the toolbar, click Search.

2. In theSearch Query field type your search criteria.

3. Select a Time from the list to further narrow the results.

4. Select a specific Repository or choose to show all data.

5. Click Search.


86 Chapter 3: Usage: Action


Search TermsImmediate Insight search works similarly to an Internet search engine. Enter a list of terms, separated byspaces, into the search box to find all the records that match those terms.

Key Concepts

l An unstructured search works best when you start with as little specificity as possible. Enter a fewwords (fragments of an error message, names of a server or user, etc.) and see the results it returns. That will often catchmore related information than if you start out with amore detailed query.

l The goal of a search is first to get in the ballpark and find things that *might* be what you are lookingfor or might be related, even if they aren’t exact. Because results come back quickly, you can thenrefine your search and have a better chance of not missing important or interesting related inform-ation.

l Use of quotationmarks and wildcards are very useful. Terms are often tied together with unexpecteduse of spacing and punctuation. but if your search isn’t returning the results you were expecting, tryadding or removing a wildcard or quotationmarks to capture a different result.

l The value of streaming data tends to decline rapidly over time, somost of the focus of ImmediateInsight is on what is happening right now. Searches tend to generate themost useful results whenyou start by looking at what has happened in the last hour or last 24 hours rather than trying to look ata report for yesterday or a week ago.

l To select a term that wasn’t pre-highlighted, you can copy and paste or double-click after selectingfrom the Search Action bar.

Qualifiers

By default, the search with several terms will look for records containing any of those terms. You can useAND, OR and NOT to createmore complex and specific searches.Qualifiers before search strings can beused to tell Immediate Insight more specifically what metadata to search, for example text:string,source:string, tag:string.

Qualifier Example of Use

ANDUse to search for records containing two terms.

Fred AND login

NOTUse to search for records containing only one term and not another.

Fred NOT login

OR A search forMike JoeMary, generates the same result as MikeOR JoeOR Mary.

+Plus is equivalent to AND. Results should contain the term.

+vlan2

-Minus means results NOT containing the term.

-vlan2

SearchQualifiers

Chapter 3: Usage: Search Terms 87

Grouping

Use parenthesis to logically group things for easier understanding.

(FredMary Joe) AND login NOT Server32

QuotationMarks

Use of quotationmarks make a search termmore specific. This “192.168.0.1” will yield amore specific resultthan this 192.168.0.1; which could return hits containing any of those four numbers.

As a general rule, if your search is not returning the results you were expecting, try the search again using quo-tationmarks.

Wildcards

Wildcards are very useful for cases like object names or domain names. Wildcards are also useful for partialsentries.

l Asterisk (*)—using an asterisk in place of a partial entry will return significantly more results thanwithout the asterisk. For example, unstructured logs will frequently usemultiple forms of a verb orterm. For example, start* is helpful to find both phrases like “started service foo at 03:12:19” and“03:12:19 service foo starting up”.

l Tildes (~)—can be used to find similar words, one or two characters difference, such as amisspellingor pluralization.

l Regular Expression (/ /)—terminology can also be used. Place the regular expression insideslashes.

l IP address range—the range can be within a sub-net, or across contiguous sub-net ranges.

Character Examples of Use

*192.168.0.*

Finds 192.168.0.1 and 192.168.0.254. It also finds 192.168.0.1:8080,192.168.0.1/8080, and 192.168.0.1.8080

~error~

Finds “error” and “errors” (and also themisspelled version “eror”)

/ //Server[6-8]/

Finds “Server6”, “Server7”, or “Server8” but not “Server4”. Some regular expressions,such as lookarounds and greedy matches, can be quite slow or resource intensive.

IP address range ipv4:[10.1.1.1 TO 10.3.100.252]

Useful Wildcards for Searches


88 Chapter 3: Usage: Grouping


Metadata

By default, the search will look for matches across all data or metadata. Recordmetadata includes location,source, and other correlations. You can view metadata for an event record by hovering over it and selectingmetadata from the inlinemenu. For example, to find all records from any syslog collector, just search for sys-log.

To find all records from the file collector that you named Log Collector:

“Log Collector”

Metadata can be identified by field names, although because everything in an unstructured system isindexed together, there is often no benefit in specifying a field name.

To use a field name, put the field name in front of your term, followed by a colon. This would find all recordswhosemetadata source string contains the word “syslog” but would not capture records with the word syslogin the text of themessage:

source:syslog

The following would capture records with the word “syslog” in the text of themessage but would not capturerecords that are part of a syslog collector:

fulltext:syslog

Time

The default time range for a search is the last 60minutes. Keep inmind that this is sort of relative time spe-cification is a continuously moving window, so if you do the exact same search a few seconds later, thecount of matches may not be identical.

You can select a rough time specification from the time drop down next to the search box. You can enter aspecific time range with the “custom” time selection, but with unstructured data, themost effective way tonavigate time is usually to start from a larger period and then zoom in by clicking bars on the time sliceschart.

Start from last hour, last day, or last 7 days, and then zoom in to the precise time period you are interested in. Often you will see interesting anomalies as you go, such as patterns of previous occurrence of the sameevent, anomalous dropouts or spikes in activity, or timestamps that are slightly off from what you expecteddue to configuration issues.

Chapter 3: Usage: Metadata 89

About the Microscope PageTheMicroscope page is used to view presentation of detailed data. It detects and displays common data inan easy-to-read format. It includes JSON and JSON fragments, XML, FireEye Security Appliances and PaloAlto Networks Security Appliance.

TheMicroscope provides the greatest level of detail about a particular event. Depending on the source dataandmetadata enrichment, some events will show more details than others. You can link to theMicroscopefrom a Search Result.

There are three options to provide a different perspective of the data.

l Fulltext displays a complete view of the record.

l Metadata displays additional details about the record.

l Origin Info displays information about where the record came from and how it is stored.

To select a different perspective, click its name or use theView Selector buttons.

Click to move to the previous or next event in the list.


90 Chapter 3: Usage: About theMicroscope Page


About the Reputation PageThe Reputation page allows you to add what you have learned about a particular host(s). Such ‘human’insights are automatically correlated and searchable alongside ‘machine’ data to provide amore completepicture.

For an entity, you can create a custom key value pair. It becomes part of the reputation for the entity and isadded to subsequent matching event and entity combinations. Example key value pairs include, CriticalInfrastructure: Oracle; hasPriviledge:Twitter; isExec:CFO.

There are four options to provide a different perspective of the data.

l Reputation Info

l Associations

l Activity

l Observations

To select a different perspective, click its name or use theView Selector buttons.

Chapter 3: Usage: About the Reputation

Add Reputation InformationTo add reputation data, complete the following steps.

1. On the toolbar, click Reputation > Reputation Info.

2. Next toReputation Info, click theAdd icon.

3. Enter text in theEntity, Field, Value andExpires fields.

4. Click Save.

Note: In addition to themethod above, bulk reputation info can be imported via CSV.

Add Bulk Reputation InformationAdding bulk Reputation information can be imported into Immediate Insight using a .csv file.

You can drag files ending with .iprep or .iprep.csv toDataFlow > Import > Import as Lines to populate theIP reputation fields.

The first columnmust be IPMATCH and contain IPv4match patterns. Fields are taken from the column head-ings and the values from each row.

An event is generated for each entry so that other actions, workflows can be tied to changes.

Overlapping reputations are allowed. Non-conflicting fields aremerged in. Conflicting fields are overwritten toallow easy updating by reloading the IP reputation files again.

To add bulk reputation information using a .csv file, complete the following steps.

1. Create the .csv file.

Note: IPMATCH must always be the first heading and contain IP addresses or subnets.

2. Save as a comma delimited CSV file.

3. Go toDataFlow > Import, and drag the .csv file into Import as Lines.

4. A dialog box will open, Import file in progress... notifying you of the import progress, click Dismissto close.

5. You can go toSearch to view the successful import.

6. Go toReputation to view the imported information.

a. Enter an IP address from the .csv file.

b. Click Lookup.

c. Click theAdd icon to includemore reputation information.

d. Click Save.


92 Chapter 3: Usage: Add Reputation Information


Delete Reputation EntriesDeleting reputation entries is performed by setting the first value to IPDELETE.

IPMATCH,Restricted

3.3.4.0/24,IPDELETE

2.2.2.20 to 2.2.2.21,IPDELETE

Events are generated for every delete:

Caution!A deletion will remove the entire reputation including fields that weremerged in from otherfiles.

Pay special attention when converting .xls files to .csv files as some data files may contain multiple delim-iters whichmust be removed in order to produce a clean csv file.

Chapter 3: Usage: Delete Reputation Entries 93

About Pinboard and BookmarksPinboardPinboard is another term for bookmark. It provides quick access to favorite, often-used searches and queries.Pinboards are useful starting points to monitor and investigate known issues or suspected scenarios.

You access your pins from the Home page by clicking thePinboard icon. Click a title to open its relatedpage.

To change the information that is displayed, click a pinboard icon. The available icons and their function arelisted in the following table.

Icon Name Description

Pinboard Click this icon to open a list of available Pinboards.

Counts Displays event counts for a selected time period. Click thearrow to choose a time period.

Change Defines the time frame for comparison. Click the arrow tochoose a time period.

Concise view Click to display only counts.

Category view Click to display all category items on the same graph.

Expanded view Click to display graphs and other details. One entry perline.

Graph view Click to display only graphs.

Questions Click to display a list of questions pertaining to the searchresults.

Pinboard Icons

BookmarksYou can access bookmarks from the toolbar at any time by clicking theBookmark icon.

l Add to Pinboard or Bookmarks opens a dialog box used to add new pins or bookmarks.

l Manage Pinboard & Bookmarks opens a dialog box used to hide, share or delete bookmarks.

l Show History opens theSearch History dialog box which displays a list of themost recent querysearches.

l Share a Session allows you to join an existing session or begin a new one.


94 Chapter 3: Usage: About Pinboard and Bookmarks


l Share a Configurationopens the Shareboard, where you can view current items available to beexported and shared.

Add a Pin or BookmarkTo add a new pin or bookmark, complete the following steps.

1. There are two ways to add a pin or bookmark.

a. On the toolbar, click Search. After entering your search criteria, click the pin icon next tothe search box.

b. Click theBookmarks icon on the toolbar and select Add to Pinboard or Bookmarks.

2. Complete theAdd to Pinboard or Bookmarks dialog box.

a. TheSearch Terms field is populated with the actual query used for the search.

b. TheDisplay Name field is based on the search. You can leave as is or change it.

c. TheDisplay on Pinboard check box is selected by default. Clear the check box to remove itfrom the pinboard.

d. Select aSection from the list or leave blank.

e. Select aPage from the list or leave blank.

f. Select theDisplay on Bookmarks Menu to add the search there as well.

g. Select theUse as a shortcut in search (auto-replacement) check box if this query will beused often.

h. Click Save orMore Details to add additional details about the pin.

More Details

i. In theSharing section, enter an optionalManagement Name andDescription,

j. In theDisplay section, select Panel Types from the lists.

k. In theColors section, select colors for various areas of the search.

l. In theQuestions section, typeQuestions and select Actions.

m. Click Save.

Manage Pinboard and BookmarksTomanage existing pinboards and bookmarks, complete the following steps.

1. On the toolbar, click theBookmarks icon and then click Manage Pinboards & Bookmarks.

2. From the list of current bookmarks, select the one tomanage.

3. On theAdd to Pinboards or Bookmarks dialog box, make your changes to any of the fields.

4. Click Save.

Chapter 3: Usage: Add a Pin or Bookmark 95

Share a SessionShare a Session allows you to join an existing session or begin a new one.

To share a session, complete the following steps.

1. On the toolbar, click theBookmarks icon and then click Share a Session.

2. In the Live Share dialog box, enter a tag to join an existing session or begin a new one.

3. Select the Follow this search activity stream orPublish my search activity for others to followoption.

4. Click OK.

Quit SharingTo quit sharing, clear the field and then click OK.

ShareboardThe Shareboard downloads a copy of individual remote commands, workflows, and bookmarks for you toshare with others.

Click theShareboard icon or from theManage Pinboards & Bookmarks dialog box, click Share.

To share, click Export to create a data file of the included items. This data file can be imported into other sys-tems.


96 Chapter 3: Usage: Share a Session

Chapter 4: API

APIs: Getting Data Out 98

REST API Certificates 101

Command-Line Client 102


APIs: Getting Data OutBesides interactive search and the export button, there are a variety of programmatic ways to get data out ofthe system. These include:

l HTTPS Query

l Linux command line client, for use in scripting

l Custom Feeds in the data router

HTTPS Query APIThe Immediate Insight REST API lets you query the search engine from scripts or other products by sendingURLs.

A basic query looks something like this. You can type it into your browser to try it and see the results:

https://ip-address:3201/search?k=api-key&q=query

Note: See the REST API Examples section below for more examples.

Note that:

l Queries are sent over HTTPS to port 3201 with a REST endpoint of /search .

l The API Key functions as a password.

l You can definemultiple API Keys.

String Function

q search term (default = all records)

n number of search results to return (default = 10)

endtime end time as either a JavaScript time integer or ii human readable form(default = "now")

starttime end time as either a JavaScript time integer or ii human readable form(default = "1 hour ago")

time length of time as either a JavaScript time integer or ii human readableform (default = "1 hour") overrides starttime

k API Key (default = none , note: use “api-key” to match the pre-con-figured API Key in API Credentials)

f format text|json|html|csv (default = text)

d type of data info|events|summary|full|timeline (default = events)

type the field to return summary or event info about (default = entities, full-text)

API Parameters


98 Chapter 4: API: APIs: Getting Data Out


REST API ExamplesGet the 10most recent records:

https://ip-address:3201/search?k=api-key

Get the 10most recent records with the word root:

https://ip-address:3201/search?k=api-key&q=root

Get the 10most recent records from the logfile "firewall.log":

https://ip-address:3201/search?k=api-key&q=sourceFile:firewall.log

Get the 50most recent records from the logfile "firewall.log":

https://ip-address:3201/search?k=api-key&q=sourceFile:firewall.log&n=50

Display them in a browser-readable format:

https://ip-address:3201/search?k=api-key&q=firewall.log&n=50&f=html

Return them in a spreadsheet-readable format (csv):

https://ip-address:3201/search?k=api-key&q=firewall.log&n=50&f=csv

Return them with additional metadata in a software-friendly format (json):

https://ip-address:3201/search?k=api-key&q=firewall.log&n=50&f=json

Display all records matching the search "error" for the last 2minutes (up to 250):

https://ip-address:3201/search?k=api-key&q=error&n=250&starttime=now-2m&endtime=now

Display the last 10 records matching the search "error" over the last hour:

https://ip-address:3201/search?k=api-key&q=error&starttime=now-1h&end-time=now&d=events

Display just the count of records matching the search "error" over the last hour:

https://ip-address:3201/search?k=api-key&q=error&starttime=now-1h&end-time=now&d=info

Return the count of records matching the search "error" eachminute over the last hour (suitable for a graph):

Chapter 4: API: REST API Examples 99

https://ip-address:3201/search?k=api-key&q=error&starttime=now-1h&end-time=now&d=timeline

Return both the event text and themetadata for the last 10 records matching the search "error" over the lasthour:

https://ip-address:3201/search?k=api-key&q=error&starttime=now-1h&end-time=now&n=10&d=full

Return a count for each of the top 100 entities in that search:

https://ip-address:3201/search?k=api-key&q=error&starttime=now-1h&end-time=now&n=100&d=summary

API CredentialsThe REST endpoint for API Queries is /search. The default port number is 3201 and protocol is HTTPS.

HTTPS access must be enable in Immediate Insight CLI before API calls can bemade.To enable HTTPSaccess, complete the following steps.

1. Type set-ssl command to enable encryption on browser sessions.

2. Type reload server to make changes take effect.

3. Close the browser and reopen using HTTPS instead of HTTP (https://ip-address-of-server:3201)

Note that:

l The API key functions as a password.

l You can havemultiple API keys active at once.

l The default API key is "api-key".

API Keys are defined in app/config/marshal-settings.conf . You can edit this file to add or changethem. Options include:

l Default—API.key = "api-key"

l NoKey Required—API.key = none

l Specify an API key (quotes are optional)—API.key = my-secret-key

l Set several API keys—API.key = ["key1","key2","key3"]

l Disable the REST API entirely—API.enable = false (default value is true)

l Restart themarshal for changes to take effect by using the restart command—restart


100 Chapter 4: API: API Credentials


REST API CertificatesBecause REST API calls are over HTTPS, it is suggested to enable certificate keys.

CertificatesFireMon recommends the best practice use of matching CA certificates installed in user’s browsers toreduce the possibility of man-in-the-middle attacks and provide a smoother user experience.

During installation, a self-signed rootCA pair is generated automatically in app/config/certs.

Note: You can replace this pair with your ownCA by overwriting the rootCA.key and rootCA.pem files,however this is an advanced task. Most can use the self-signed certificates provided.

To set the certificate, complete the following steps.

1. Type set-certs followed by reload server to activate the certificate.

2. Copy the app/config/certs/rootCA.pem file from the Immediate Insight server to your computerusing an SFTP client.

3. Load the Certificate into your browser.

A. In the Chrome browser go toSettings > Show Advanced Settings > HTTPS/SSL >ManageCertificates > Trusted Root Certification Authorities > Import.

B. Specify the rootCA.pem file.

4. Restart the Chrome browser.

The next time you log into Immediate Insight you should not see a certificate warning.

Note: While the system has a reasonable set of security measures in place, the present release isdesigned to run in a secure and trusted environment. If you have a need to expose it directly to the Inter-net, please call us to discuss additional hardening procedures.

Chapter 4: API: REST API Certificates 101

Command-Line ClientThe command-line client (CLI) lets you issue search queries from a Linux shell script or terminal session. You can run the script from the Immediate Insight console and also download it to remote systems.

The CLI uses the REST API. See the API Credentials section for details on how to create API keys. Thedefault API key is "api-key".

InstallationTo put the command prompt client script on another system or expose the script on your own ImmediateInsight system, you first download it from the appliance.

1. From the Linux shell, download either of the following scripts:

l wget --no-check-certificate https://ip-address:3201/tools/ii

l curl -o ii -k https://ip-address:3201/tools/ii

It is intentionally designed as a simple shell script so that you can customize or modify it foryour needs. You will need to have the Curl HTTP client utility installed on your system. Mostfull Linux distributions include this by default and also in their repositories.

sudo yum install curl or sudo apt-get install curl

2. Mark the script as executable:

chmod +x ii

3. Edit the first lines of the script (using vi or nano) to set your server address and API key (if you wish).

For convenience, the default server address and default API key are set in the first few lines of the file. Youcan edit it andmodify these for your own environment to eliminate the need to specify them on every query.

l KEY—"api-key" (substitute other API key if you are using a different one)

l SERVER—"localhost" (substitute IP address of II server if accessing remotely)


102 Chapter 4: API: Command-Line Client


Command-Line Options

String Function

-h Print a list of options

-q The search query [default=*]

-n Number of results to return [default=10]

-t Time scope [default="now"]

-a Start time [default="now"]

-b End time [default="now"]

-f Format [text|json|html|csv] [default=”text”]

-d Data type [info|events|summary|full|timeline] [default=”events”]

events Returns just the original text of the event

full Returns the event text and also themetadata such as source, arrival time, and cor-relations

info Returns just a count of matching records

timeline Returns count-per-minute data over the time period

summary Returns a list of entities and their counts

-s Server address [default="localhost", or whatever you substituted when editing ii script]

-k API Key [default="api-key", or whatever you substituted when editing ii script]

Command-Line ExamplesTime examples

"now", "now-1h", "now-1d-4h-5m" "1361683874554"

Get the first 10 results matching the query from server 192.168.0.1 with credential “api-key”

ii -s 192.168.0.1 –k api-key -q "search terms" -n 10

If you already set the default address and api key for your server and placed the ii file in your executablepath, this would shorten to:

ii -q "search terms" -n 10

Chapter 4: API: Command-Line Options 103

You can pipe or direct the output in typical unix shell fashion. For example, this would get themost recent100 lines from the source “fw.log”, split fields based on colon, select the value from the third field, and providea count of each unique value:

ii -q "fw.log" -n 100 | cut -d ":" -f 3 | sort | uniq –c

Get just a count of the number of hits:

ii -q "search terms" -d info

Get just a count of the total number of hits for the last 5minutes:

ii -q "search terms" -d info -a now-5m -b now

Get just a count of the number of hits for the last 5minutes, on a per-minute basis

ii -q "search terms" -d timeline -a now-5m -b now

Get a list of the entities matching the search, and the number of hits for each:

ii -q "search terms" -d summary

Get the full data (message + entities + correlations + source, etc.) for themost recent 20 records matchingthe search:

ii -q "search terms" –n 20 -d full


104 Chapter 4: API: Command-Line Examples

Immediate Insight User's Guide€¦ · application initialized datastore initialized Do you want...

Documents

Transcript of Immediate Insight User's Guide€¦ · application initialized datastore initialized Do you want...