Operational Intelligence Webinar | SQLstream | July 2013 Series

Copyright © SQLstream Inc.

Analytics, Predictive Analytics, Prescriptive Analytics:

The Anatomy of Operational Intelligence

The SQLstream July 2013 Webinar series

| 2Copyright © 2013 | +1 877 571 5775 | [email protected]

• Deliver webcasts that explain real-time Big Data and the world of Operational Intelligence in real terms

• Introduction to streaming data management for real-time, high velocity, low latency applications

• Share our thoughts, experiences, industry use cases and examples

MISSION


• July 9 2013 |10:00am PSTAnalytics, Predictive Analytics, Prescriptive Analytics: The Anatomy of Operational Intelligence

• July 16 2013 |11:00am PSTListen to your Sensors: A Tale of Managing Large Scale Sensor Networks in Real-time

• July 23 2013 |11:00am PSTPredict and Avert: Using Log File Data to Prevent Cybersecurity and Fraud Attacks in Real-time

• July 30 2013 |10:00am PSTNo more CPR for your CDRs: Meet Real-time Traffic Utilization, Billing and Fraud Detection

The Operational Intelligence Series


Analytics, Predictive Analytics, Prescriptive Analytics: The Anatomy of Operational Intelligence

As the speed of business accelerates, more data is created every second.

Gaining a competitive edge relies on transforming massive volumes of log file and other machine-generated data into actionable intelligence in real-time.

Leading organizations are leveraging real-time operational intelligence to make smarter decisions faster, and at price they can afford.


SQLstream, Inc.

History Launched 2009

Over 1.5M lines of code

Multiple deployments across many industries, with top real-world benchmarks

Features Supports all forms of

unstructured and structured data

Accelerates and extends Hadoop & RDBMS

Not limited to SQL

Key innovations Only true streaming

data management platform

Only true standard SQL streaming engine

Covered by 5 broad patents for stream processing

A streaming data management platform for real-time Operational Intelligence from high-velocity Big Data.

Operational Intelligence


Bridging The Chasm

As we move toward a real-time business environment, the capability to process data flows swiftly and flexibly will become increasingly important. SQLstream leads the industry in this kind of capability.

“

”Robin Bloor

Chief Analyst for Bloor Group

Aberdeen’s research has shown that Best-in-Class organizations are demanding access to actionable intelligence faster than ever. This is precisely the growing demand that SQLstream is meeting with their Streaming Big Data Engine, while continuing to bring other attractive features like full Hadoop integration.

“

”Nathaniel Rowe

Leading Analyst for Aberdeen Group

Business Intelligence

Post-hoc Analysis

Data Warehousing

Strategic insights

Operations

Transaction

Processing

Machine Data

Everyday business

Operational Intelligence integrates Operations and BI

Operational IntelligenceOptimizes tactical decisions from real-time actionable

insights

Combines operations data with BI data continuously

Provides Real-time integrated view of the business and

operations

SecurityCompliance

FraudQuality

PromotionAdvertising

Cross-selling

SecurityCompliance

FraudQuality

PromotionAdvertising

Cross-selling


The Information Value Chain

What is happening?

What might happen?

What just happened?

Make stuff happen!


Drivers For Operational Intelligence*Source: Ventana Research Operational Intelligence Benchmark Research, 2013

Analyze complex relationships

“Most organizations do not have technologies that can analyze complex relationships across multiple source systems in real-time, but most are planning to deploy”

Low latency is essential“No latency can be tolerated for security

and network management for example”

Integration is the missing link

“Most Business Intelligence systems are not integrated with operational intelligence data”

Specialized tools perform better

“When organizations use specialized tools they are much more satisfied with the Operational Intelligence initiatives”

Manage riskManage risk

Manage performance

Manage performance

Comply with regulations

Comply with regulations

Identify opps. for improvement

Identify opps. for improvement

59%

59%

58%

58%

54%*Importance of goals in current or planned deployment of technology


Machine-generated Big Data ExplosionHigh volume, high velocity, structured and unstructured data from software platforms, applications and systems

GPSGPS

TelematicsTelematics

IP Networks, VideoIP Networks, Video

Servers, Social Media, SecurityServers, Social Media, Security

Servers, Applications, Storage Servers, Applications, Storage

NetworksNetworks

Machine-generated data will increase Machine-generated data will increase to 42% of all data to 42% of all data by 2020, up from by 2020, up from 11% in 2005.11% in 2005.““The Digital Universe in 2020”The Digital Universe in 2020” IDCIDC


Connecting Machine Data, Connecting The World

DATA SOURCESDATA SOURCESINDUSTRIESINDUSTRIES


Machine DataWhere is the intelligence?

TRANS,2013-02-17-15:30:22,3458783,2347897953,128.56.0.253,STATUS:-15, DE69975, 4157588342Transaction Log Details

Web Server Logs

CDR Records

Smartphone GPS Updates

Twitter

{"created_at:Thu Feb 17 15:30:55 +0000 2013,id:304612775055998976,id_str:304612775055998976,text:@MyServiceProvider today sucks, keeps dropped!,source:u006ca href=http:www.url.com rel=nofollow,followers_count:147,friends_count:10142, location: San Francisco, time_zone: Pacific, geo_enabled:true, location:u00dcT: -6.1987552,106.8661953, screen_name:APerson

<id>1597831220</id><deviceid>0198873465</deviceid><lat>lat=47.643957</lat><lon>lon= -122.3269</lon><time>2013-02-17T15:37:26Z</time><bearing>223.4535</bearing>

<id>1597865781</id><deviceid>0198873465</deviceid><lat>lat=47.645982</lat><lon>lon=-122.327500</lon><time>2013-02-17T15:37:26Z</time><bearing>200.6138</bearing>

<id>1597940125</id><deviceid>0198873465</deviceid><lat>lat=47.647381</lat><lon>lon=-122.326501</lon><time>2013-02-17T15:37:26Z</time><bearing>87.4357</bearing>

[Sun Feb 17 15:30:49 2013] [notice] srv-sfo-08 caught SIGTERM, shutting down[Sun Feb 17 15:30:49 2013] [notice] Apache/2.2.21 -- resuming normal operations

TERMINATE,ctl09gsx,01299796304,GMT-08:00,02-17-13,15:21:00,9,387,64ms,02-17-13,15:30:55,0005, IP-TO-IP,4157588342,8775715775,1,0,4157588342,RD_AXY_NN0_001,SFR01AAG34,40.50.245.60, 234.234.60.75,65678,411,399,SIP,SANFRANCISCO,0x4B1698,0x0005E,0x49768,4157588342,0198873465

Timestamp

Timestamp

Timestamp

Timestamp

Timestamp

Timestamp

Timestamp

Timestamp

Timestamp

Timestamp

Mobile #Mobile #CustomerCustomer

Mobile #Mobile #Device IDDevice IDTerm

ReasonTerm

Reason

Device IDDevice ID LocationLocation

LocationLocation

Service ProviderService Provider

Fail CodeFail Code

ServerServer


Real-time Big Data Heat Map

Hot right now

Hot, growing fast (< 2 years)

Growing fast (2 – 5 years)

5 – 10 years

> 10 years

Total Cost of Performance


RECORDS PER SECOND

LATE

NCY

Total Cost Of Performance (total COP)The High-Velocity, Low-Latency Tipping Point for Big Data

Patterns Trends MiningConnections

Searches Inventory ReportsStatistics Billing

SOCIAL E-COMM SECURITYTELEMATI

CSTELECOM

TradingAdvertisin

gAlerts

Detection

Signal

Intelligence

TO

TA

L C

OST


Intelligence

TELECOM

Patterns Trends MiningConnections

Searches Inventory ReportsStatistics Billing

TradingAdvertisin

gAlerts

Detection

Signal

SOCIAL E-COMM SECURITYTELEMATI

CS

RECORDS PER SECOND

TO

TA

L C

OST

LATE

NCY

Total Cost Of Performance (total COP)The High-Velocity, Low-Latency Tipping Point for Big Data

Streaming Data Management


High-velocity Big Data Analytics

Historical queries and data

enrichment

Storing valuable derived streams for future access

Op

era

tion

al In

telli

gen

ce

Continuous Queries over Sliding Time Windows Analysis and Integration of Unstructured and Structured

data Prescriptive Analytics drives Automated Actions


Real-time ArchitectureStreaming Analysis and Integration for Infinite Flows of Unstructured Data in Real Time

Streaming Agent & Adapter Layer + JDBC APIHadoop Streaming

Query Planner & Optimizer for MPP ExecutionSQL

Developer Tools

Platform Administration

Streaming SQL Real-time Applications

Real-time Dashboards & Visualization

Impala SQL

HBase

HDFS / MR

Hadoop for Stream Persistence, Enrichment & Replay (Optional)

Any external data warehouse, operational system and enterprise

platform


CLEANING & CLEANING & FILTERINGFILTERING

CLEANING & CLEANING & FILTERINGFILTERING

STREAMING STREAMING ANALYTICSANALYTICS

STREAMING STREAMING AGGREGATIONAGGREGATION

CONTINUOUSCONTINUOUSINTEGRATIONINTEGRATION

Internet of Internet of EverythingEverythingFraud Fraud

PreventionPreventionNetwork Network

MonitoringMonitoring

CybersecurityCybersecurity QoS andQoS andQoEQoE

An Operational Intelligence Platform


SELECT STREAM ROWTIME, url, numErrorsLastMinute FROM ( SELECT STREAM ROWTIME, url, numErrorsLastMinute, AVG(numErrorsLastMinute) OVER lastMinute AS avgErrorsPerMinute, STDDEV(numErrorsLastMinute) OVER lastMinute AS stdDevErrorsPerMinute FROM ServiceRequestsPerMinute WINDOW lastMinute AS (PARTITION BY url RANGE INTERVAL ‘1’ MINUTE PRECEDING) )

AS S WHERE S.numErrorsLastMinute > S.avgErrorsPerMinute + 2 * S.stdDevErrorsPerMinute;

The Power Of Streaming SQL

BUSINESS NEED:Predicting run-away applications before

resource consumption becomes an issue.

BLAZING SPEED:Processing millions of

records per second on low-end servers.


The SQLstream s-Streaming Product Portfolio

s-ServerData Management Platform for Streaming Big Data

s-AnalyzerReal-Time Visualization for Streaming

Operational Intelligence

s-TransportGeo-Analytics for Location-based

Applications

s-VisualizerAdvanced

Visualization

s-Clouds-Server EC2 AMI Deployment

Industry Use Cases


Real-time Operations – IT And BeyondHigh Velocity: The Next Frontier For Big Data

INTERNET OF INTERNET OF THINGSTHINGS

NETWORKS & NETWORKS & SERVICESSERVICES

SECURITYSECURITY

Machine-to-MachineMachine-to-Machine

Cars as SensorsCars as Sensors

Web SecurityWeb Security

Banking & FinanceBanking & FinanceTelecommunicationsTelecommunications

Customer LoyaltyCustomer Loyalty


Log Analytics For Security Intelligence

Identity Theft Monitoring

» Real-time detection and prevention

» Real-time log and web feed analysis on a massive scale

Fraud prevention

» Identify and block suspicious account activity

» Monitor transaction and activity logs in real-time

Cybersecurity Attacks

» Identify patterns of activity for Advanced Persistent Threats

» Log file and security device monitoring, with geospatial analytics


Internet Of Everything – Use Case Examples

Intelligent Transportation

» Reduce congestion and travel times; improve traveller experience

» Real-time flow and congestion prediction from GPS data

Telematics (V2V Infrastructure)

» Reduce ‘walkaway’ events and warranty costs

» Real-time vehicle health monitoring every 10 ignition cycles

M2M

» Monetization of M2M data feeds

» Real-time wireless sensor analytics and aggregation


Telecommunications – Use Case Examples

CDR Analytics

» Real-time charging and QoS / QoE monitoring

» CDR collection, session reconstruction and analysis at scale

4G Wireless Network Performance

» Identify and prevent QoS exceptions

» 4G cell performance data analysis in real-time


DATA EXPLOSION

COMPLEXITY

BUSINESS AGILITY

Streaming Operational Intelligence

Eliminates the development risk•Simplifies development, rapid time to market

Lowest Cost of Performance for Real-time Apps•Efficient scale-out for high velocity data

Adding new applications on the fly•With dynamic sharing of data streams across Apps

Damian Black

Email | [email protected]

Website | www.sqlstream.com

Operational Intelligence Webinar | SQLstream | July 2013 Series

Technology

Transcript of Operational Intelligence Webinar | SQLstream | July 2013 Series