Big Data, Analytics and Real Time Event Processing
Transcript of Big Data, Analytics and Real Time Event Processing
VP Developer Evangelism Samisa Abeysinghe
Big Brother for Enterprises: Big Data, Analytics and Real Time
Event Processing
12 Feb 2014
2
About the Presenter
๏ Samisa AbeysingheVP Developer [email protected]
๏ Samisa Abeysinghe, Vice President of Developer Evangelism joined the company in September 2005. Prior to the current role, Samisa used to be VP of Engineering and managed the development of WSO2 Carbon based product platform.
3
About WSO2๏ Global enterprise, founded
in 2005 by acknowledged leaders in XML, web services technologies, standards and open source
๏ Provides only open source platform-as-a-service for private, public and hybrid cloud deployments
๏ All WSO2 products are 100% open source and released under the Apache License Version 2.0.
๏ Is an Active Member of OASIS, Cloud Security Alliance, OSGi Alliance, AMQP Working Group, OpenID Foundation and W3C.
๏ Driven by Innovation
๏ Launched first open source API Management solution in 2012
๏ Launched App Factory in 2Q 2013
๏ Launched Enterprise Store and first open source Mobile solution in 4Q 2013
4
What WSO2 Deliver
NSA Monitoring
๏Notorious for the kind of surveillance they do
๏But did you think of the
๏ Data volumes involved
๏ Correlation complexities in analytics
๏ Off line vs Real time operations implications
๏ Some operations got to be in real time, else the value is lost
๏ Summarized data over time (and other) dimentions for analytics
5
Why should I bother?๏ Deal with terra bites of information
๏ In order to make decisions and take action
๏ Real time
๏ Offline
6
Data Sources and Information Dimensions
Data Sources / Storage(Volumes in Terra Bytes)
Vertical Correlation
(Across Same Source)
Horizontal Correlation(Across multiple sources)
Time Dimension๏Batch processing
๏ Off-line, not real time
๏ Useful for analytics (BAM)
๏ Business intelligence
๏Real time
๏ Event correlation (CEP)
๏ Simple (vertical)
๏ Complex (horizontal)
๏ Insights as it happens
๏ Same time actions/ reactions
Fundamental Technologies Involved
๏Data Collection, Analytics & Presentation
๏ Business activity monitoring (BAM)
๏ Real time summary & historic summary
๏Pattern matching, in real time
๏ As and when it happens
๏ In event driven mode, complex event processing (CEP)
๏ Identify meaningful events
๏ Opportunities or threats
9
WSO2 :ONLY COMPLETE & INTEGRATED Platform๏Complete and integrated for
๏ Data capture
๏ Analysis: both real-time and batch
๏ Visualization
๏ Action taking business process execution
Making data driven intelligence for your enterprise easy
10
WSO2 Big Data Analytics Platform for Your Enterprise
WSO2 Big Data Analytics Platform for Your Enterprise
Key Elements
Data Collection
Data Analysis
Data Visualization
Taking Action
What will you have with WSO2 for Big Data? ๏A complete pre-integrated solution for all
your big data needs
๏Without needing to integrate together other packages from different vendors
๏This can save you valuable time in
๏ Getting up and going in quick time
๏ Giving you the most comprehensive solution
๏ Capable of scaling from small, low cost to almost unlimited size
๏ Data gathering and analysis problems
13
Business Activity Monitor (BAM)
๏Can capture data from any application/service
๏ APIs, Services or source including file-based data
๏WSO2 has lightweight agents to provide
๏ raw data from numerous sources
๏ as well as support for
๏ gateways (lightweight or heavyweight)
๏ Service and API proxies
๏ Data imports from various sources (including file based data)
๏ Any other source (extended agents)
14
Complex Event Processing (CEP) • CEP process multiple event streams generated
within the enterprise to construct data abstraction and identify meaningful patterns among those streams
• Powered by WSO2 Siddhi• Processes more than 2.5M events/sec on single server (commodity hardware)
• Has a powerful and extensible SQL Like Siddhi query language
• Integrates with any enterprise system for event capture
• Support multiple transports & event formats: HTTP, JMS & SMTP on XML, JSON, Text & Map
• Efficient large volume event publishing to enterprise systems and custom notifications
• XML, JSON, Text over HTTP, JMS & SMTP
• SMS Support
• MySQL DB and Cassandra Support
16
Other Big Data tools by WSO2
๏Data Services Server
๏ can front almost all databases or no-SQL databases and provide raw data gathering/exposing services
๏Enterprise Mobility Management (EMM)
๏ Provide mobility monitoring allowing you to capture information about mobile application and mobile device usage
๏Social Media Adapters (extension points)
๏ Allow you to collect data from major social media sites
18
Putting Them All Together
๏Together, the WSO2 toolset gives you
๏ The capability to capture the raw data
๏ from almost any source you need
๏ to make more intelligent analysis and decisions
๏ of yours and your customers’ activities
19
BAM & CEP Combined
๏A combined way to get both batch delayed Hadoop style analysis
๏Use big data based analysis and visualization tools to compute KPIs
๏Look for trends and incidents
๏BAM allows you to
๏ Calculate systematic data in real-time
๏ CEP allows you to
๏ find complex event sequences with ultra-fast performance in real-time
20
BAM & CEP Provides
๏Compute averages, moving averages
๏Other statistical values
๏Detect sequences of actions that constitute a suspicious or interesting behavior
๏ Then for you to take action on those
๏You can replay time and try different combinations
๏ Detect what are key sequences that may be important to watch for
21
Take Action
๏Using CEP, BAM and Business Process Server you can ๏ Trigger complex actions from any analysis short-
circuits,
๏ Quick action hits,
๏ Liquidations,
๏ Defection alerts,
๏ Escalation policies
๏ Advantage: offer solutions your customers cannot refuse.
WSO2 platform gives you the complete solution to take action from the
intelligence you gather22
Visualize
๏Use Jaggery.js, User Engagement Server or dozens compatible visualization tools you can ๏ Present the raw data and analysis
๏ Monitor the business processes in real time
๏You can configure mashups, graphs, heat-maps that show you both historical data and real time data interactively
๏You can initiate business processes and monitor business processes from your dashboards you create
23
24
Visualize & Help Decide
25
Business Model
26
Call to action page
๏ Be your own NSA: http://wso2.com/landing/nsa/
๏ Big Data Webinar Series: http://wso2.com/landing/wso2-bigbrother-webinar-series/
๏ WSO2 Business Activity Monitor: http://wso2.com/products/business-activity-monitor/
๏ WSO2 BAM Docs: http://docs.wso2.org/display/BAM240/WSO2+Business+Activity+Monitor+Documentation
๏ WSO2 Complex Event Processor: http://wso2.com/products/complex-event-processor/
๏ WSO2 CEP Docs: http://docs.wso2.org/display/CEP300/WSO2+Complex+Event+Processor+Documentation