#SDF16

Building High-scalable Enterprise Solutions with Microsoft Azure and PowerBIThomas Claudius Huber – Principal Consultant@thomasclaudiush

#SDF16

Thomas Claudius Huber

Principal Consultant

– Microsoft Most Valuable Professional (MVP)

Focus on C#, TypeScript, Angular and Azure

– Part of Cloud-Unit @Trivadis

Book- and Pluralsight-Author

– See also www.thomasclaudiushuber.com

#SDF16

Agenda

The Big Picture

Azure Event Hubs

Stream Analytics and Power BI

#SDF16

The Big Picture

#SDF16

The big picture

#SDF16

Agenda

The big picture

Azure Event Hubs

Stream Analytics and Power BI

#SDF16

Azure Event Hubs

#SDF16

Azure Event Hubs

Scales up to millions of events

Great approach for telemetry data

Great approach for sensor-data

#SDF16

#SDF16

Azure Event Hubs: Partitions

Event stream is segmented by partitions for scale

– Scaling for event consumers

– Scaling for event submitters

You can have between 2 and 32 partitions

Azure Support can enable up to 1024 or even more «under special conditions»

#SDF16

Azure Event Hubs: Throughput Units

Throughput units (TUs) are important for billing. 1 Unit means

– Write maximum 1MB/sec and 1000 messages per second

– Read maximum 2MB/sec

Number of partitions has to be equal or greater than number of TUs

TUs are applied on namespace level

– You can have up to 10 event hubs in a namespace

– Namespace is used for the URI, for example https://<yourNamespace>.servicebus.windows.net

#SDF16

Azure Event Hubs: Consumer Groups

Views on the Event Stream

– Similar to topic subscriptions that are also a «view» on an event stream

– Organizational anchor for checkpointing

$Default consumer group is always there

– Up to 20 named consumer groups are supported

Limitations

– 5 concurrent reading connections per consumer group

• It’s recommended to use a separate consumer group for each job

#SDF16

Azure Event Hub vs. Azure IoT Hub

Quote: «It is not uncommon to use both IoT Hub and Event Hubs in the same solution»Official docmentation: https://azure.microsoft.com/en-us/documentation/articles/iot-hub-compare-event-hubs/

Event Hub IoT Hub

Communication Device to cloud Device to cloud and cloud to device

Security Event hub wide shared access policy Per device identity

Scale 5000 simultaneous connections Millions of simultaneous connections

SDKsSupported on .NET and C, also has AMQP and HTTP (REST) interfaces

Device SDKs for several platforms.

#SDF16

But why not always using IoT Hub?

IoT Hub scales on devices

– a lot of concurrent connections

– but is expensive for a few thousand devices and a lot events

Event Hub scales on events

– but is limited to 5000 concurrent connections

#SDF16

So what about pricing?

Event Hub

CHF 0.0253 per million events

1 Throughput Unit = CHF 0.0135/hr (~CHF 10/month)

– 1 Throughput Unit = 1MB / 1000 Messages Ingress, 2MB read

#SDF16

Creating an Event Hub: Step 1

Go to the https://portal.azure.com

Select «New Internet of Things Event Hubs»

Enter Namespace-Name and Resource Group and click «Create»

#SDF16

Creating an Event Hub: Step 2

So far you’ve created a namespace

– the namespace can have up to 10 event hubs

When you navigate to the namespace (via «Resources»), you can add a new event hub

#SDF16

Demo: Create an Event Hub

#SDF16

Connect to the Event Hub

To push events to an event hub, you have several options:

– From a .NET application you can use the NuGet-packageWindowsAzure.ServiceBus and the EventHubClient-class

• Samples: https://github.com/thomasclaudiushuber/EventHub.Clients

– You can also use a REST-API that is available automatically for each event hub

• https://github.com/thomasclaudiushuber/EventHub.RestClientGenerator

You need to setup a Shared Access policy for your event hub

To access the event hub, you need the URI and the shared access key

– both is part of a Connection String that you find in the Shared Access policy

#SDF16

Pushing Events to the Event Hub from a .NET App

Use the NuGet-package WindowsAzure.Service

– It has an EventHubClient-class

Events are JSON-data items

#SDF16

Consuming Events from the Event Hub in a .NET App

Same EventHubClient-class

Grab the consumerGroup and create the receivers for the partitionIds

#SDF16

Demo: Access the Event Hub

#SDF16

Agenda

The big picture

Azure Event Hubs

Stream Analytics and Power BI

#SDF16

The big picture

#SDF16

What is Stream Analytics

it’s a real-time event processing engine

Use a SQL-variant to query the event stream

High scalable

#SDF16

Stream Analytics: Inputs and Outputs

#SDF16

Stream Analytics: Windowed Aggregates

There are different windowed aggregates, one is the Tumbling Window

select DeviceName,max(value),count(*)into [SDFPBOutput]from [SDFEventHub99] Timestamp by Readtime group by devicename, TumblingWindow( Duration(second, 5), Offset(millisecond, -1))

#SDF16

Demo: Setting up Stream Analytics

#SDF16

Power BI as an Output

BI-tool to visualize your data

– Access it via https://www.powerbi.com

Stream Analytics can directly push its outupt to Power BI

In Power BI you can work with the Stream Analytics Data

#SDF16

Build a Live Dashboard in Power BI

#SDF16

Agenda

The big picture

Azure Event Hubs

Stream Analytics and Power BI

#SDF16

The big picture

#SDF16

QuestionsThomas Claudius HuberPrincipal Consultant

thomas.huber@trivadis.com@thomasclaudiush