Real-Time Business Intelligence with Microsoft SQL Server 2008 R2 Jesus RodriguezChief Architect, TellagoMicrosoft Architect AdvisorMicrosoft MVPOracle SOA ACE

SESSION CODE: BIE403

About me…

Chief Architect, Tellago, IncMicrosoft advisorMicrosoft MVPOracle SOA ACESpeaker, authorhttp://weblogs.asp.net/gsusx

Agenda

The challenges of real time business intelligenceIntroducing Microsoft StreamInsightReal Time BI patterns Demo, Demo, Demo, Demo, Demo

Real time business intelligence?

Real-time Business Intelligence systems are event driven, and use Event Stream Processing techniques to enable events to be analysed without being first transformed and stored in a database.

These in- memory techniques have the advantage that high rates of events can be monitored, and since data does not have to be written into databases data latency can be reduced to milliseconds.

Capabilities?

Event Driven

Continuous queries

Time Sensitive

Low Latency

High Volume

Event processing is the foundation of real time BI…

Scenarios?

Financial Sector

Call Centers

RFID Systems

Web Analytics

What are the challenges of traditional (batch) BI in real time scenarios?

Query Overloading

Agility

ETL Dependencies

Has the ETL run yet?

Report Abuse

Ok I got it…..How do I implement something like that on the Microsoft

platform?

Real time event processing was the missing piece….

Introducing Microsoft StreamInsight

Microsoft StreamInsight is a platform that you use to develop and deploy complex event processing (CEP) applications. Its high-throughput stream processing

architecture and the Microsoft .NET Framework-based development platform enable you to quickly implement robust and highly efficient event processing applications.

Microsoft StreamInsight

27

CEP EngineO

utput Adapters

Input Adapters

Event

Standing Queries

Event sources Event targets

`

Devices, Sensors

Web servers

Event stores & Databases

Stock tickers & News feeds

Event

Event

Event

Event

Event

Event

C_ID C_NAME C_ZIP

Event stores & Databases

Pagers & Monitoring devices

KPI Dashboards, SharePoint UI

Trading stations

Event

Event

CEP Application at Runtime

Static reference data

IDE

.NETC#

LINQCEP Application Development

Developing Your First StreamInsight Application

DEMO

Events

Events expose different temporal characteristicsPoint in time eventsInterval events with fixed durationInterval events with initially unknown duration

Rich payloads capture all properties of an event

t1 t4t3t2 t5Time

Payl

oad/

val

ue

ab

c de

Event Types

Events in Microsoft’s CEP platform use the .NET type systemEvents are structured and can have multiple fieldsFields are typed using the .NET framework types CEP engine provisioned timestamp fields capture all the different temporal event characteristicsEvent sources populate time stamp fields

Timestamps/Metadata

LongpumpID

StringType

StringLocation

Doubleflow

Doublepressure

… … … … … …

Event Streams & AdaptersA stream is a possibly infinite sequence of events

Insertions of new eventsChanges to event durations

Stream characteristics:Event/data arrival patterns

Steady rate with end-of-stream indication Intermittent, random, or in bursts

Out of order events: Order of arrival of events does not match the order of their application timestamps

AdaptersReceive/get events from the data sourceEnqueue events for processing in the engine

31

LINQ Query Examples

LINQ Example – GROUP&APPLY, WINDOW:

from e3 in MyStream3group e3 by e3.i into SubStreamfrom win in SubStream.HoppingWindow( FiveMinutes,ThreeSeconds)select new { i = SubStream.Key, a = win.Avg(e => e.f) };

LINQ Example – JOIN, PROJECT, FILTER:

from e1 in MyStream1join e2 in MyStream2 on e1.ID equals e2.IDwhere e1.f2 == “foo”select new { e1.f1, e2.f4 };

Join

FilterProject

Grouping

Window

Project &Aggregate

Time Windows

Time

Tumbling Window

Time Windows

Time

Hopping Window

StreamInsight Hosting Models

CEP in-process server (explicit model)

Application

CEP explicit logic

CEP Server

Application

In-process server modeHosted server mode

Real Time BI Patterns with Microsoft StreamInsight

Real Time Distributed Event Processing

UI applicationStreamInsight Server

QueryStart Query

Events

UI application

Start Query

EventsOutput Adapter

Input Adapter

The event processing logic leaves on the StreamInsight ServerClients are able to start and stop queriesMultiple clients subscribe to the same even types

Real Time Distributed Event Processing

DEMO

Aggregations and Windowing

UI application StreamInsight Server

QueryStart Query

Events

Output Adapter

Input Adapter

The aggregation formulas are calculated in the StreamInsight ServerAggregations are processed using different time windowsThe logic can be extended with custom aggregates

Real Time Event Windows and Aggregations

DEMO

Real Time Event Replay

UI application StreamInsight Server

QueryReplay

Events

Output Adapter

Input Adapter

The StreamInsight server tracks the events processed by the different input adaptersThe client is able to replay events based on specific time windowsThe StreamInsight server uses a replay adapter to reprocess specific events

Event tracking

Real Time Event Replay

DEMO

Self Service BI + Real Time Event Processing

PowerPivot UI StreamInsight Server

Query

Output Adapter

Input Adapter

The StreamInsight server tracks the events processed by the different input adaptersThe PowerPivot UI polls the data into its own databaseThe user can use the data to calculate specific aggregations

Event tracking

PowerPivot UI

Self-Service Event Analysis with StreamInsight and PowerPivot

DEMO

Conclusions

Event processing is the foundation of real time business intelligenceMicrosoft StreamInsight enables complex event processing capabilities on the Microsoft platformTake into account real time business intelligence patterns such as time windows aggregations, replays, event subscriptions, etcTechnologies like Microsoft PowerPivot can complement the capabilities of StreamInsight with self-service data mechanisms

Resources

www.microsoft.com/teched

Sessions On-Demand & Community Microsoft Certification & Training Resources

Resources for IT Professionals Resources for Developers

www.microsoft.com/learning

http://microsoft.com/technet http://microsoft.com/msdn

Learning

Complete an evaluation on CommNet and enter to win!

Sign up for Tech·Ed 2011 and save $500 starting June 8 – June 31st

http://northamerica.msteched.com/registration

You can also register at the

North America 2011 kiosk located at registrationJoin us in Atlanta next year

© 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to

be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.