SQL Server 2008 R2 Data Warehousing

SQL Server 2008 R2 Data Warehousing

Ciprian Jichici

General Manager Genisoft Microsoft Regional Director Romania

Agenda

• Review: Key S+S Data Platform Requirements• SQL Server 2008 R2: The Data Platform for S+S• Warehousing – A Classic Approach• Introducing StreamInsight

− Choosing a Development Model− Choosing a Deployment Model

• Master Data Services

Review: Key S+S Data Platform Requirements

• Comprehensive

• Flexible• Global

Storage and Search

• Responsive• Available• Secure• Manageable

Federation

• Coordination• Consolidation• Transformation• Synchronization

Composition

• Monitor• Analyze• Plan• Collaborate

Insight

• On client• On premise• Partner-hosted• Microsoft-hosted

Delivery

• Productive• Interoperable• Supports key standards

Data Programmability

SQL Server 2008 R2: The Data Platform for S+S

Database Engine

Relational Data

Management

Replication

Full Text Search

Integration Services

ETL Processing

Data Profiling

StreamInsight*

Complex Event

Processing

Analysis Services

Classic OLAP Data Manageme

nt

Data Mining

PowerPivot* Self Service

Analytics

Reporting Services

Managed Reporting

Self Service Reporting

Embedded Reporting

Master Data Services*

Master Data

Management

* New offering

Warehousing• Fundamentals• SQL Server Scalability and High Performance for

Warehousing• Clustering• Backup and Restore• Database Mirroring• Database Snapshots

Warehousing – A Classic Approach

Demo

SQL Server 2008 R2: Storage and Search

• New application scenario for SQL Server• Process high volumes of event stream data with

low latency• Compliments existing solutions

Complex Event Processing (CEP)

• Integrated development platform for CEP applications

• Modular, provider-based architecture• Input adapters read event streams• Event models support diverse patterns of

events• Runtime engine hosts standing LINQ queries

built in .NET• Output adapters transmit results

Introducing StreamInsight


• Concept of Complex Event Processing• Introduction to StreamInsight and Overall

Architecture• Choosing a Development Model• Choosing a Deployment Model

Complex Event Processing

• Processing and querying of event data streams• Data queried while “in flight”• May involve multiple concurrent event sources• Works with high data rates• Aims for near-zero latency

Isn’t This Just A Database Application?

Database CEPQueries Ad-hoc on stored data Continuous standing

queriesLatency Seconds MillisecondsData Rate Hundreds per Second Tens of Thousands per

Second

request

response

Eventoutput streaminput

stream

Aims of Complex Event Processing

• Identify from seemingly unrelated events− Patterns− Relationships− Gaps (Expected events that did not occur)− Abstractions

• Trigger immediate response actions

“Sweet-spot” For Complex Event Processing

Relational Database Applications

Financial Trading Applications

Aggregate Data Rate (Events/sec)

0 10 100 1000 10000 100000

higher

MonthsDays

Hours

MinutesSeconds100 ms

< 1 ms

Operational Analytics Applications, e.g., Logistics, etc.

Manufacturing ApplicationsMonitoring Applications

CEP Target Scenarios

Data Warehousing Applications

Web Analytics Applications

Usage Example: Capital Markets• Algorithmic trading• Smart order routing• Real-time profit and loss• Rapid analysis of transactional cost• Fraud detection• Risk management

Usage Example: Click-stream Analysis • Often 100,000 events per second• Automate

− Page Layout− Navigation− Presentation− Targeted Advertising

Usage Example: Communications• Real-time network monitoring• Quality of service monitoring• Provide location-based services• Fraud detection• Intrusion detection

Usage Example: Command Intelligence• Battlefield control• Monitoring of resource locations• Intrusion detection• Network traffic analysis

− Emails− Network traffic− Watch-lists− Financial movements

Usage Example: Manufacturing• Asset monitoring• Aggregation of machine-based sensor data• Generation of alerts in error conditions• Identifying the “golden batch”

Usage Example: Casino Monitoring• Gaming machine event analysis• Card table analysis

− Fraud detection− Profit and loss in real-time

• Targeted advertising− Player behavior− Loyalty system implementation

Usage Example: MPG and Virtual Worlds• Real-time monitoring• Managing player interest• Web-site traffic analysis• Detecting and eliminating undesired behaviors• Understanding behavioral patterns

Usage Example: Public Health• Patient management• Outbreak management• Trend detection• Insurance risk analysis

Usage Example: Logistics• Vehicle management• Supply chain forecasting and tracking• Maritime logistics• GPS tracking

Usage Example: Energy Management• Monitoring

− Consumption− Variations

• Detecting outages• Smart grid management• Aggregating data across the grid

Competitive Landscape

Industry Forum: http://complexevents.com

Progress – Apama IBM – Open ESBWest Global – Total Insight Coral8Bristol Technology – SenActive GmbH – Real Time Monitoring

Streambase Agent LogicInetco TibcoOracle – Fusion Middleware Event ZeroSyndera Rulecare

Microsoft StreamInsight

• Platform for development and deployment of CEP applications

• High-throughput stream processing architecture• .NET based development environment

StreamInsight Aims

• Monitor data from multiple sources and detect− Meaningful patterns− Trends− Exceptions− Opportunities

• Analyze data without storing it first• Low latency• Trigger response actions• Mine events for new business KPIs

Benefit: .NET Development Environment• Use .NET languages such as C#• Query using LINQ• Take advantage of developer familiarity with .NET• Reduces development times (and costs)• Extend StreamInsight with .NET code

Benefit: Performance and Data Throughput

• Highly parallel execution platform• In-memory caches• Incremental result computation• All processing triggered by incoming events

− Avoids polling overhead• Can handle out-of-order events• Can incorporate static reference or historical data

Benefit: Deployment and Management

• Multiple deployment scenarios− Fully integrated via embedded DLL− Stand-alone server (multiple apps and users)

• Monitoring and management built-in− Management interface− Diagnostic views− Manageability framework allows for remote monitoring− Stand-alone event flow debugger

StreamInsight Event Data Flow

Data Sources, Operations, Assets, Feeds, Sensors, Devices

Monitor &

Record

Operational Data Store &

Archive

CEP Enginef(x)

g(y)

CEP Enginef(x)

f'(x)

g(y)

h(x,y)

Histor

y

Deploy

Resultsf'(x

)h(x,y

)

Manage&

Benefit

Mine&

Design

Input Data Streams

Input Data Streams

OutputData Streams

StreamInsight Architecture

StreamInsight Deployment Alternatives

Data Sources

Aggregation & Correlation

CEP

CEP

CEP

CEP for lightweight processing and filteringCEP for aggregation and correlation of in-flight eventsCEP for complex analytics including historical data

Devices

Sensors

Web servers

Feeds

CEP

CEP CEP

CEP

CEP CEP

CEP CEP CEP

Complex Analytics &Mining

End to End SolutionDemo

Demo Scenario: Highway Monitor

• Major highway

• 8 Lanes• 2 Directions• 6 Toll-Points• Vehicle Types

− Car− Bus− Truck− Ambulance− Taxi

Toll-Point Timings• Vehicles are measured multiple times• Electronic tag captured• Speed is measured• Vehicle type is determined

Toll-Point Measurements• EventID (guid)• TollPointId (which toll-point) – 0 to 5• DirectionId (which direction) – 0 (North) or 1 (South)• Lane (which lane) – 0 to 7 • VehicleTypeId – Emuneration

− (0-car, 1-bus, 2-truck, 3-taxi, 4-ambulance)• TagId – vehicle’s individual tag• EnterGate – datetime when vehicle entered gate• MillisecondsToPassSpeedCheckPoint

− How long did vehicle take to travel 10m• ExitGate – datetime when vehicle exited gate

Agenda



StreamInsight Development Models• Three development models

− Implicit Server− Explicit Server− IObservable/IObserver

Implicit Server Development Model

• Easiest development model to use• Hides most complexity• StreamInsight server acts as an implicit host

− Creates and registers required objects• Allows developers to focus on query logic

− Good test environment for query developers

Implicit Server: Process

• Server created implicitly for each query• Query is automatically hosted

− Stored in memory, not on stable storage (disk)• Developer provides:

− Input/output adapter bindings− Query logic and binding

• Server manages objects− Creates event type (accesses it via reflection)− Registers input/output adapters− Registers query template and query− Creates objects when query started− Disposes objects when query is disposed

Implicit Server: Query Example

var inputConf = new TextFileInputConfig("MyData.csv");var outputConf = new TextFileOutputConfig("Result.csv"); var inputStream = CepStream<PayloadType>.Create("inputStream", typeof(TextFileInputFactory),inputConf, EventShape.Point); CepStream<PayloadType> filter = from e in inputStream where e.value > 10 select e; Query query = filter.ToQuery(typeof(TextFileOutputFactory), outputConf, EventShape.Interval, StreamEventOrder.FullyOrdered);query.Start();// perform work or wait for output adapter to signal completequery.Stop();

Explicit Server Development Model

• Developer explicitly creates and registers all objects• Provides complete control of

− StreamInsight application− Development environment (via client-side object model API)

• Allows for reuse of− Queries− Adapters− Event Types− 3rd party Query Templates

Explicit Server: Process

• Code creates or connects to server• Code creates all objects and registers them into

server− Query Template− Query Instance− Event Types− Input/Output Adapters

• Server can store metadata in memory or persist to disk

• Server can run locally or remotely via web service

Explicit Server: Query Example 1 of 2

Server server = Server.Create();Application application = server.CreateApplication("app1"); InputAdapter ia = application.CreateInputAdapter<MyInputAdapterFactory>( "InputAdapterName", "Some description");OutputAdapter oa = application.CreateOutputAdapter<MyOutputAdapterFactory>( "OutputAdapterName", "Some other description"); CepStream<PayloadType> inputStream = CepStream<PayloadType>.Create("inputStream"); var highWaterMarkAlert = from e in inputStream where e.value > 99 select e;



Explicit Server: Query Example 2 of 2

QueryTemplate qt1 = application.CreateQueryTemplate( "highWaterMarkAlert", highWaterMarkAlert);QueryBinder qb = new QueryBinder(qt1); qb.BindProducer("inputStream", ia, new MyInputAdapterConfig(), EventShape.Interval);qb.AddConsumer("queryresult", oa, new MyOutputAdapterConfig(), EventShape.Point, StreamEventOrder.FullyOrdered); Query q1 = application.CreateQuery( "query1", qb, "Query Description");q1.Start();

Explicit Server: Connect To Existing Server

// Connect to a remote CEP server.Server server = Server.Connect( new System.ServiceModel.EndpointAddress( @"http://localhost:12345/CEP")); // Retrieve the application app1 from the server.Application application = server.Applications["app1"]; // Retrieve existing query template, qt1, from server.QueryTemplate qt1 = application.QueryTemplates["qt1"];

IObservable/IObserver Development Model

• Alternate to using Input/Output adapters• Observer is an object that wishes to be notified

when another object’s state changes• Observable is an object whose state changes can

be requested via registration• Avoids the need to define temporal properties of

events• Works well with existing event-driven applications

− Use StreamInsight to provide real-time analysis and insight

IObservable/IObserver: Process

• Application must implement − IObservable class for event producer− IObserver class for event consumer

• StreamInsight server is configured to use these classes

• Queries are created and bound

IObservable/IObserver: Example Code

// Connect to a remote CEP server.Server server = Server.Connect( new System.ServiceModel.EndpointAddress( @"http://localhost:12345/CEP")); // Retrieve the application app1 from the server.Application application = server.Applications["app1"]; // Retrieve existing query template, qt1, from server.QueryTemplate qt1 = application.QueryTemplates["qt1"];

Agenda



StreamInsight Deployment Scenarios

• Full integration into host application as embedded DLL

• Stand-alone server− Multiple applications − Multiple users− Can be packaged as a Windows Service

Deployment: Hosted DLL

• Choose this option for any of these scenarios:− Using Implicit Server development model− Relying on IObservable/IObserver programming paradigm− Need to minimize memory footprint− Wish to use a single process for both application and CEP

server− Wish to tightly control access to the CEP server

• Do not choose this option if:– You require shared access

to the metadata– You require shared access

to the streaming event data

Deployment: Stand-alone Server

• Choose this option for any of these scenarios:− Metadata objects need to be shared between applications

− Event Types− Adapter Types− Query Templates

− A data source registered with the server provides an event stream for another existing application

• StreamInsightHost.exe can be the server host

Deployment: Other Potential Scenarios

• StreamInsight-enabled device

• Building operational intelligence

Master Data ServicesDemo

SQL Server 2008 R2 Data Warehousing

Documents

Transcript of SQL Server 2008 R2 Data Warehousing