Data to Insight: Introduction to WSO2 Business Activity Monitor

Anjana FernandoSenior Technical LeadWSO2 Inc.

Outline● Introduction to Business Activity Monitoring

● WSO2 BAM v3.0: Next Generation Analytics

● WSO2 BAM High Availability Distributed Deployment

● WSO2 BAM Solutions

Business Activity Monitoring

WSO2 BAM v3.0

● What’s new?

○ Faster analytics with Apache Spark, 10x - 100x speedups

○ Rich indexing support

○ Pluggable data stores, from light-weight embedded RDBMS to highly scalable HDFS

○ Embeddable architecture for inclusion with other Carbon servers

Data Agents● Compatible with CEP/BAM

● Get data across to BAM

− Service monitoring feature – WSO2 AS, DSS, ESB, API Manager

− Mediation monitoring feature – BAM Mediator for WSO2 ESB

− Custom data-agents

● Asynchronous & non-blocking

● Multiple data transport support with Thrift, Kafka, JMS, MQTT and more.

● BAM analytics specific REST service for index operations, data retrieval and search

Data Receiver● Receives data and stores it in the backend data store

○ Pluggable data store, ranging from light-weight databases to highly scalable big data stores

● Asynchronous & non-blocking

○ Combination of Cassandra, Thrift and the non-blocking nature results in extremely fast writes

● Shared with WSO2 CEP for real time analysis

● Supports plugging in of different receiver types

Data Model● Data is sent over using strongly typed, versioned data streams

{ 'name':'phone.retail.shop', 'version':'1.0.0', 'nickName': 'Phone_Retail_Shop', 'description': 'Phone Sales', 'metaData':[ {'name':'clientType','type':'STRING'} ], 'payloadData':[ {'name':'brand','type':'STRING'}, {'name':'quantity','type':'INT'}, {'name':'total','type':'INT'}, {'name':'user','type':'STRING'} ]}

● BAM Analytics store it in a backend table store structure, with optional indexing, for efficient data lookup, pagination and near-realtime search

Analytics Data Abstraction Layer● A well defined data source API for exposing storage for analytics record and index store

○ Analytics Record Store

■ A schema-less table record store for storing individual records, with timestamps and

pagination supported

○ Analytics File System

■ A file system structure implementation to be used by the data indexing

implementation

○ Initial set of connectors to be supported

■ RDBMS - with any RDBMS supported using a configuration based query templates

■ HDFS

■ MongoDB

The Analyzer Engine● Powered by Apache Spark with querying through Spark SQL

● Parallel, distributed processing with optimized in-memory computing

● Outperforms Hadoop in efficiency and speed, making smaller deployments feasible for typical analytics tasks

● Clustered using Spark embedded Carbon servers

The Analyzer Engine● Runs on a Spark engine embedded Carbon server cluster

○ Scalable analytics

○ Cluster can range from a couple of nodes to 1000s

● Analysis is carried out on an interactive query console and analytics scripts

● Queries are based on an easy-to-learn, SQL-like query language

INSERT INTO TABLE UserTable SELECT userName, COUNT(DISTINCT orderID), SUM(quantity) FROM PhoneSalesTable WHERE version= "1.0.0" GROUP BY userName;

The Analyzer Engine● Interactive Query Console

○ Queries are entered in an console and executed one by one in the cluster, and the results are

sent back asynchronously to the console

● Scripts can be scheduled

○ e.g.:- once a minute, every Wednesday at 4:15 p.m., every 30th at 12 midnight

● Connects to the standard analytics data store

○ Any type of backend data store is supported, to read data to be analyzed and resultant data can also be written out using the same interface

○ Single set of analytics scripts / toolboxes for any type of backing data store

Distributed Data Indexing● Data indexing support with full text indexing

● Data drill down support with facets

● Based on Apache Lucene, high performant, feature rich, indexing engine

● Distributed indexing support using sharded Lucene indexes

○ Horizontal scaling of index storage and indexing performance

○ Designed to be able to dynamically add more nodes later

● Near real-time data indexing and retrieval

○ Data is indexed immediately when the data is retrieved by the event receiver

○ Batched data indexing for higher performance

The Presentation Layer● Gadget based dashboards for visualization

○ Custom user dashboard and gadget creation using wizards

○ Data drill-down views

● Activity Dashboard for correlating activities

● Message Console - one stop shop for managing the analytics data store

○ Full support for paginated data retrieval and timestamp based filtering

○ Data table creation, index association, row insertion, updates, deletions

○ Batch data upload using files

○ Rich search functionality for indexed tables

● Log analysis dashboard - out-of-the box log analysis solution

The Presentation LayerBAM Dashboards

The Presentation LayerGadget Portal

The Presentation LayerMessage Console

Message Console (cont'd)

BAM Toolboxes● BAM Toolboxes are installable and hot deployable artifacts used for deploying functionalities to a

BAM server

○ Stream definitions○ Analytics scripts○ Dashboards

● Toolboxes for monitoring and auditing most WSO2 products are available OOTB

● Toolboxes for custom scenarios can be created easily

H/A Distributed BAM deployment● WSO2 BAM can be clustered and deployed in a distributed manner to enable high-availability, fail-

over scenarios

○ Distributed deployment

■ All components of BAM are clustered (data receiver, storage, analyzer and presentation)

■ Hazelcast in-memory data grids are used for clustering implementation

■ Optimized for a simpler deployment

Service Statistics Monitoring● Improved set of analytics and visualization for web services and web applications

● Available with upcoming WSO2 AS v6.0 release

● In-built dashboard in WSO2 AS with light-weight analytics module

○ Configurable to point to an external BAM server

Service Statistics Monitoring

Service Statistics Monitoring

Service Statistics Monitoring

Service Statistics Monitoring

Service Statistics Monitoring

Service Statistics Monitoring

Service Statistics Monitoring

Service Statistics Monitoring

Activity Monitoring● Activity monitoring is for tracking events from multiple nodes in a flow to understand a specific

activity

○ e.g.:-

■ A client initiating a web services request which travels through multiple ESBs,

application servers and returns back. This flow will be uniquely identified and

visualized in BAM

○ Used for tracing messages, finding performance hotspots in the flow

○ Implemented based on a correlation id based mechanism and indexing

○ Upcoming: Mediator level tracing and profiling in WSO2 ESB 5.0

Activity Monitoring

Activity Monitoring

Log Analysis● OOTB log analysis solution in BAM v3.0

● Log event indexing

○ Uses the new BAM v3.0 indexing support

○ Event attributes can be indexed to be search by server, cluster, log type and also log

messages itself for full text search

● Custom search queries using Lucene queries and regular expressions

● Log agent client for reading and publishing log events from log files

○ No need to install a special agent inside the target servers itself

○ Extensible mechanism for specifying log formats in parsing

Log Analysis

Log Analysis

Questions?

Thank you!