(NPI) v1.2

© 2016 IBM Corporation1

Network Performance Insights (NPI) - v1.2.0

Dec 8th 2016

Installation, Configuration & Integration steps forNetwork Performance Insight (NPI) – v1.2

Prepared by:Krishna M Kodali – [email protected] Parish – [email protected]



Dec 8th 2016

Agenda§ Netcool Network Management (NNM) v9.3 Quick recap.

§ NNM Extension for Network Performance Insight (NPI) v1.2

§ NPI Overview

§ NPI Architecture

§ Installation Procedure for NPI

§ Integration

§ Lab Integration walk through

§ Q&A



Dec 8th 2016

NNM v9.3 – Product Suite § NNM 9.3 product suite has same version as in NOI v1.4.0

§ More about NNM v9.3 deployment @ https://ibm.biz/BdsyW4

§ IBM Tivoli Netcool Omnibus v8.1§Netcool/OMNIbus Server (Core) Components

§Probes (Mttrapd & Syslog)

§Gateway Components (JDBC)

§Netcool/OMNIbus WebGUI

§ IBM Tivoli Network Manager(ITNM) v4.2§ Network Manager Core Components

§ Network Manager GUI Components

§ Network Manager Reports

§ IBM DB2 10.5 Enterprise Server Edition

§ IBM Tivoli Netcool Configuration Manager(ITNCM) v6.4.2§ Netcool Configuration Manager (Worker & GUI Server)

§ Drivers

§ NCM Reports



Dec 8th 2016

Lab Reference – 4 server setup

Server#1 – ITNM Core & Omnibus Core§ Netcool/OMNIbus Server (Core) Components

§Probes (Mttrapd & Syslog)

§Gateway Components (JDBC)

§ITNM Core Components

Server#2 – DB2 & DASH§ DB2 Server – v10.5 with multiple db instances

§NCIM

§ITNCM

§REPORTER

§TCRDB

§WAS & Jazz SM for DASH

§TCR

§ITNM Reports

§ITNCM Reports

§Device DASHBoard (NPI)

User & Database Information:§ Non-root user - ‘netcool’

§ Database user – ‘ncim’

§ Databases – NCIM42, ITNCM, REPORTER, TCRDB

Server#3 –ITNCM § ITNCM Worker Server

§ ITNCM Presentation Servers

§ ITNCM DriversLocal/SAMBA/NFS with all packages

Local/SAMBA/NFS with all packages

Local/SAMBA/NFS with all packages

Server#4 - NPI Cluster§ Network Performance Insights

§ Apache Ambari

Non-root Users exclusive for NPI:hdfs, mapred, yarn, ams, kafka, spark, zookeeper, netcool Note: All non-root users gets created during installation, no manual steps.



Dec 8th 2016

Network flow recordsA network flow record is data generated by a network device, such as a router or

switch

The data in a network flow record describes the network traffic that has passed through the router or switch

Network devices that have network flow enabled send their network flow records

to a collector

IBM Network Performance Insight collects these flow records, analyzes

them, and stores them

IBM Network Performance Insight users get reports and dashboards to help them

understand the traffic on their network

Network device

Network flow enabled router

Traffic sent from one router to the other

Network flow analysis

Network flow collector

Network flow record that describes the traffic that was sent

IBM Network Performance

Insight

Network administrators use IBM Network Performance Insight reports and dashboards

NPI Overview



Dec 8th 2016

NPI Overview – cont’dIBM Network Performance Insight features:

§ Supports NetFlow (Cisco), JFlow (Juniper), CFlow (Alcatel), NetStream (Huawei), and IPFIX flow records Supports IPv4 and IPv6 traffic

§ Interactive dashboards with drill-down support to network and interface level reports that show near real-time data. These dashboards are supported on Dashboard Application Services Hub portal in Jazz for Service Management

§ Traffic data that can be categorized by the domain observation points or interfaces, and can be aggregated up to 1-minute granularity

§ Integrated with IBM Tivoli Netcool/OMNIbus component of IBM Netcool Operations Insight for monitoring network traffic threshold violations

§ Integrated with IBM Tivoli Network Manager component of IBM Netcool Operations Insight for device details lookup

§ Flexible retention periods and raw and aggregated data, with the ability to balance smaller storage requirements with longer analysis period



Dec 8th 2016

Network flow records and IBM Network Performance Insight reports

Flow-enabled network devices inspect traffic and send network flow records to IBM Network Performance InsightThese flow records include the following data:§Source IP address§Destination IP address§Source port§Destination port§Layer 3 protocol§Type of service§Router or switch interface

IBM Network Performance Insight collects the network flow records, analyzes them, and presents reports and dashboards



Dec 8th 2016

Using IBM Network Performance Insight: Traffic detailsYou can access the following 15 traffic details reports by changing the report view:•Top Sources•Top Sources with Application•Top Applications•Top Applications with Source•Top Applications with Destination•Top Applications with Conversation•Top Protocols•Top Protocols with Source•Top Protocols with Application•Top Protocols with Conversation•Top Protocols with Destination•Top Conversations•Top Conversations with Application•Top Destinations•Top Destinations with Application



Dec 8th 2016

Using IBM Network Performance Insight: Domain names



Dec 8th 2016

Using IBM Network Performance Insight: Retention profiles



Dec 8th 2016

Using IBM Network Performance Insight: Thresholds



Dec 8th 2016

Traffic Details widget

Select a device

Select an interface

View flow data



Dec 8th 2016

NPI Architecture

Distributed File System (HDFS)

Persistent Message Bus (Kafka)

Batch & Streaming Compute Service (Spark)

Flow Collector

Flow Analytics UI

YARN

Dash

REST + UI Federation

NMNCIM

NM Poller

Event

Events

OMNIBus

STDIN Probe

Entity Analytics

Storage ServiceStorage, Query, FederationCalcite + Parquet + Spark

DNSBaseline Learning Threshold

Ambari

IOA Baseline Engine

ZooKeeper



Dec 8th 2016

NPI Architecture

§ Micro Service Architecture

§ IBM Open Platform

§ Decompose large system into independent services

§ Each service is isolated in it’s own process (or container)

§ Each service has well-defined responsibility

§ Each service is independently scalable

§ Each service can evolve independently from others

§ New systems can be build by composing services



Dec 8th 2016

NPI Architecture

§ 100% Apache Open Source platform

§ No-charge download

§ IOP is pre-requisite for NPI

§ Includes 22 big data components

§ NPI uses:– Ambari– Hadoop File System– YARN– ZooKeeper– Kafka*– Spark*

* NPI bundles these instead of using IOP versions, but uses IOP Ambari stack definitions

Source: http://www-03.ibm.com/software/products/en/ibm-open-platform-with-apache-hadoop

IBM Open Platform with Apache Hadoop (IOP)



Dec 8th 2016

NPI Architecture

Ambari

ProvisionaHadoopClusterAmbarihandlesconfigurationofHadoop

serviceforthecluster

ManageaHadoopClusterAmbariprovidescentralmanagementfor

starting,stopping,andreconfiguringHadoopservicesacrosstheentirecluster

MonitoraHadoopClusterAmbariprovidesadashboardformonitoring

healthandstatusofaHadoopcluster

NPI provides a streamlined Ambari stack definition

Source: https://ambari.apache.org/



Dec 8th 2016

NNM v9.3 Extension for Network Performance Insight (NPI) v1.2

§ Preparation for NPI v1.2 deployment:– NNM v9.3 must be upgraded to be compatible with NOI v1.4.0.3 level

– More information about NOI v1.4.0.3 @ https://ibm.biz/BdscMN

– Relevant Package number details @ https://ibm.biz/Bdsyd6

§ NPI Specifics:– NPI is a plug-in or rather a component

• Can not be deployed as stand-alone product– OMNIbus & ITNM are pre-requisites.



Dec 8th 2016

Installation Procedure for NPI v1.2§ Installation of NPI is 3 Phase process:

– Ensure your NNM v9.3 is upgraded NOI v1.4.0.3 level

– Ensure NPI host has met all pre-requisites - Run ‘prereq_check.sh’ which is bundled under NPI package.

– Phase#1:Ø Deploy Installation Manager (IM)

Ø Deploy NPI & Ambari via Installation Manager

Ø SSL Configuration for Single Sign-on

– Phase#2:Ø Ambari Cluster Deployment via Install Wizard

Ø Configure Managed services (e.g. NPI, HDFS, YARN etc.)

– Phase#3:Ø Install Device Dashboard v.1.1 onto DASH

ØEnable Data Flow from ITNM Poller (via Storm-Kafka) to NPI

ØEnable NPI Anomaly Thresholds



Dec 8th 2016

Upgrade NNM v9.3 to be compatible with NOI 1.4.0.3§ IBM Tivoli Netcool Omnibus Core v8.1 – Fixpack#9

§ IBM Tivoli Netcool Omnibus WebGUI v8.1 – Fixpack#7

§ IBM Tivoli Network Manager(ITNM) v4.2 – Fixpack#1

§ IBM Tivoli Network Manager(ITNM) v4.2 – Fixpack#1

§ IBM Tivoli Network Health Dashboard(NHD) – Fixpack#1 (if exists)

§ IBM Tivoli Netcool Configuration Manager(ITNCM) v6.4.2.2 (fp2)

§ Jazz for Service Management (DASH) – from 1.1.2.1 to 1.1.3.0

§ IBM WebSphere Application Server(WAS) - from 8.5.5.7 to 8.5.5.9

§ IBM Tivoli Common Reporting Services(TCR) – from 3.1.2.1 to 3.1.3.0



Dec 8th 2016

Upgrade NNM v9.3 to be compatible with NOI 1.4.0.3§ Jazz Upgrade must be done via JazzSMgrpModeUpgrade.sh (not via Installation Manager i.e. IBMIM or imcl )

§ More information about Jazz 1.1.3.0 is @ http://www-01.ibm.com/support/docview.wss?uid=swg21985946

§ If you are upgrading from DASH(JazzSM) v1.1.2.1, and if you haven’t installed in first place ‘Security Services’ – you’ll need to re-install GUI server. This isn’t necessary if you are doing a fresh deployment of DASH v1.1.3.0

§ Jazz SM 1.1.3.0 deprecated ’Security Services’

§ ‘Security Services’ Package will be installed automatically when you deploy NPI Device Dashboard v1.1



Dec 8th 2016

NPI Installation§ NPI Specifics:

– Redhat 7.2 Only

– Root Installation Only

– Libraries - libstdc++6 & lib32stdc++6 for npi-event which calls nco_p_stdin probe.

– Disable SELinux , Firewall and IPv6

– SSH passwordless login from Ambari Server to Ambari agent hosts, DASH & Ambari Server itself.

– Complete list of NPI host requirements @ https://ibm.biz/BdsyDw

– FQDN is a requirement for all the nodes in the Ambari managed cluster



Dec 8th 2016

NPI Installation – Phase#1§ NPI Specifics:

– Deploy Installation Manager

– Deploy NPI• Add IOP Folder (don’t untar IOP 3 files, download all 3 folders in same folder)



Dec 8th 2016

NPI Installation – Phase#1 cont’d§ NPI Specifics:

– Installation complete following tasks:

– Installation of Ambari Server (default port 8080)

– Installation of Network Performance Insight Repositories

– Automate the settings for communicating with DASH for Single Sign-on.• Users and Groups are created.

• Users are granted the roles.



Dec 8th 2016

NPI Installation – Phase#1 cont’d§ NPI Specifics: (Manual SSL Configuration for Single Sign-on)

§ Upon completion of NPI Installation , log onto NPI host and navigate to $NPI_HOME/installer-tools folder.§ Default NPI_HOME=/opt/IBM/npi

§ cd /opt/IBM/npi/installer-tools

§ Modify the install.User.cfg to suit your deployment & execute npiDashIntegration.sh

§ /opt/IBM/npi/installer-tools/npiDashIntegration.sh install.User.cfg default.cfg

§ This step will apply certs, create required users, groups and apply roles in ObjectServerrepository via an SSH tunnel to the DASH server.

§ As the script ‘npiDashIntegration.sh’ executes, watch the results live @§ on the NPI Server, tail the log /tmp/npiDashIntegration.log

§ on the DASH Server, tail the log /tmp/enableDash.log



Dec 8th 2016

NPI Installation – Phase#1 cont’d§ NPI Specifics: (Applying SSL configuration to DASH) Ø Upon completing the NPI installation, the administrator must apply the certs to the SSL configuration on DASH

Ø This is mandatory step whether you do DASH SSL setup via Installation Manager or manually using ‘npiDashIntegration.sh’

Ø Log onto WAS Console (default https://hostname:16316/ibm/console/logon.jsp) using ‘smadmin’

Ø Navigate to Security -> SSL certificate and key management

Ø Click on ‘SSL configurations’ under ‘Related Items’

Ø Click on ‘NodeDefaultSSLSettings’

Ø Select ‘netcool’ for both ‘Default server certificate alias’ and ‘Default client certificate alias’

Ø Apply , Save changes and Restart DASH.



Dec 8th 2016

NPI Clusters Installation via Ambari – Phase#2§ Deploy NPI Clusters via Ambari Wizard

– Open a browser & Log Onto to Ambari Console http://NPIhost:8080 - (admin/admin)

– Click on Launch Install Wizard • Enter Cluster Name• Select ‘BigInsights 4.2 NPI’ Stack from the Stacks page.• List all nodes in the managed cluster (specify one node per line)

− If all is under one server, specify FQDN of NPI Host (e.g. npi.ibm.com)

– Select 'Provide your SSH Private Key' to automatically register hosts and click SSH Private Key link on Host Registration Information pane. For root user, the private key file is /root/.ssh/id_rsa (You may have to copy the file to regular folder this file if you can’t view hidden folders).

– Click Register and Confirm.– Verify that the correct hosts for your cluster are located successfully on the Confirm Hosts page.



Dec 8th 2016

NPI Clusters Installation via Ambari – Phase#2 cont’d§ Deploy NPI Clusters via Ambari Wizard

– on the Assign Masters page, assign the master services to the hosts in the cluster and click Next.• The default assignments is available.

– on the Assign Slaves and Clients page, assign the slave and client components to the hosts in the cluster, and click Next.• User can select one or more components next to a selected host.

– On Customize Services pane, update the configuration settings for the following services . • There is a set of tabs where users can manage configuration settings Network Performance Insight components.• Note: Default recommended values are completed automatically.

− Customizing HDFS settings

− Customizing YARN settings

− Configuring communication with Jazz for Service Management− Configuring communication with Tivoli Network Manager

– Once the settings and configuration of the services are completed, click Next.– Click Deploy on the Review page.

• See the progress of the installation on Install, Start, and Test page.– The progress bar at the top of the page gives the overall status and the main section of the page gives the status for each

host. • Click Next after the services are installed successfully.• Review the completed tasks on the Summary page and click Complete



Dec 8th 2016

NPI Clusters Installation via Ambari – Phase#2 cont’d§ Deploy NPI Clusters via Ambari Wizard

– On Customize Services pane, NPI settings are key for successful Integration with NCIM & DASH. • There is a set of tabs where users can manage configuration settings Network Performance Insight components.



Dec 8th 2016

NPI Clusters Installation via Ambari – Phase#2 cont’d§ To validate the Installation navigate to /tmp look for *.log files

§ Log in to Ambari server as follows: http://<npihostname>:8080 (admin/admin)– Verify all services and components are up running



Dec 8th 2016

NPI Clusters Installation via Ambari – Phase#2 cont’d§ Managing Ambari Server, Agent and Ambari Managed processes:

– If you need to run Ambari Server and Agent as non-root, follow the steps @ https://ibm.biz/BdsMr4– More about how to stop/start Ambari managed processes can be found @ https://ibm.biz/BdsMrH

§ Manager Service CLI via npi-cmd– Displays the status of all or selected services.– Start/Stop the selected services

• /opt/IBM/npi/npi-manager/bin/npi-cmd –service npi-ui –host npi.ibm.com {start|stop}



Dec 8th 2016

NPI Installation - Device DASHBoard – Phase#3• Device DASHBoard v1.1. Installer performs a number of tasks:

• Deploys the Device Dashboard and associated widgets.

• Creates the relevant npi.properties (/opt/IBM/netcool/precision_gui/profile/etc/tnm/npi.properties)

• Installs DASH Security Services if required (JazzSM v 1.1.3.0 no longer ships ESS package)

• Creates the Console Integration for NPI UI Federation

§ Post Install steps:– The roles (noi_npi, noi_npi_admin) must be applied to the relevant users

• for e.g. user ‘npiadmin’ must be assigned with ‘noi_npi_admin’ role.

– Login as ‘npiadmin’

– Verify Poll definitions and define NPI Anamoly Thresholds

– Ensure ‘Store’ poll data option is enabled.



Dec 8th 2016

NPI Installation - Device DASHBoard – Phase#3 cont’d• Post Install Steps – Cont’d

• Login as ‘npiadmin’ , navigate to ‘Console Settings’ -> Console Integration

• Click on ‘Test’ to validate connectivity to NPI UI

• Upon successful connection, you’ll notice new ‘snowflake’ icon.



Dec 8th 2016

NPI Installation - Device DASHBoard – Phase#3 cont’d§ Post Install steps cont’d:

– Navigate to ’Poll Definitions’ and set ‘NPI Anomaly Thresholds’

– Ensure ’Store’ data option is enabled under Poll Policy



Dec 8th 2016

ITNM Integration steps & sanity check for Apache Storm – NCIM:• Overview of Data Flow

• This is done on Sever#1 where ITNM Core & Apache Storm processes are running

• ITNM Storm is a Kafka Client. It is responsible for exporting data (MonitoredObject, MonitoredInstance, PollData) to NPI via Kafka.

• Only stored polls are read via ITNM Storm Kafka and sent to NPI via Kafka.

• Simple enablement via a single configuration file kafka.properties($NCHOME/precision/storm/conf/kafka.properties)

• This file gets added as part of ITNM-FP1 deployment.

• Update following two parameters in kafka.properties:

• kafka.consumer.bootstrap.servers=npihostname:6667

• kafka.producer.bootstrap.servers=npihostname:6667

• Restart Storm - itnm_stop {stop|start}

• Data flow path ITNM (Storm) -> Kafka Broker -> NPI

• To validate if ITNM is sending its poll data to Kafka use the ncp_storm_validate.sh script ($NCHOME/precision/scripts/ncp_storm_validate.sh)

• ncp_storm_validate.sh config ( Print all properties read in)

• ncp_storm_validate.sh kafkaimport -topic nm.monitoredinstance (Listen on a Kafka topic, topics are configured in kafka.properties)



Dec 8th 2016

UI

Storage

Aggregation Engine

Data flow from NCIM to NPI via Kafka client

Network

Discovery

StormSpout

Kafka Connect

Network Interface Importer

Spark

TrafficDetails

NCIM

NCPOLLDATA

PollerMetric

Producer

Network Manager (NM) Network Performance Insights (NPI)Kafka

Kafka

NCIM

ENTITY METRICS

RAW

ITNM Collector

Entity Analytics

Polldata

Publish • MonitoredObject• MonitoredInstance• Polldata

Consume from Kafka Topics: Bulk Write

HDFS / ParquetMonitoredObjectMonitoredInstance

Polldata



Dec 8th 2016

As a ITNM Customer – why should I be interested in NPI ?• Two key benefits:

• Native functionality of Flow (Layer4 traffic) i.e. Bandwidth Monitoring.

• Detailed (granular) traffic i.e. Web, Mail, Ftp etc..

• Able to store historical raw data from ITNM Poller for longer periods (New)

• ITNM v4.2 is designed to store raw data only for an hour, rest is aggregated.

• Anomaly threshold incidents (New )

• ITNM is used for RCA, Topology Visualization & Monitoring Network

• Monitoring Via SNMP reveals total amount of Traffic

• Layer2, Layer3 statistics – such as, a link IfInErrors and IfOutErrors etc.

• ITNM Calculator for System Sizing and Poll data Storage using magic spread sheet @ https://ibm.biz/BdsMyT

• NPI Knowledge Center @ https://ibm.biz/BdsMKE



Dec 8th 2016

Questions?

(NPI) v1.2

Documents

Transcript of (NPI) v1.2