Real-time data analysis with Azure Stream Analytics
Transcript of Real-time data analysis with Azure Stream Analytics
Real-time data analysis with Azure Stream Analytics
Michael Johnson
What is streaming data?
Batch data Streaming data
Where do we find streaming data
What is Azure Stream Analytics and why would you want to use it
• What• Real-time event processing engine that helps to unlock deep insights from
data.
• Why• Easy to use
• Scalable
• Reliable, Repeatable and quick to recover
• Low cost
Azure stream analytics architectureAzure stream analytics
Inputs Outputs
Input source
• Streaming Data• Event hubs
• Blob storage
• IOT Events hubs
• Reference data• Blob storage
Output sinks
Azure data lake
Power BI
Azure SQL database Azure document DB Azure Blob storage
Azure Service bus
Azure Event hubAzure table storage
Query language overview
• DML Statements• SELECT• FROM• WHERE• GROUP BY• HAVING• CASE• JOIN• UNION
• Date and time• DATEPART
• DAY
• MONTH
• YEAR
• DATEDIFF
• DATEADD
• Aggregation functions• SUM
• COUNT
• AVG
• MIN
• MAX
• Scaling functions• WITH
• PARTITION BY
• Windowing functions• WITH
• PARTITION BY
• String functions• LEN
• CONCAT
• CHARINDEX
What does a query look like?
Tumbling windowsTUMBLINGWINDOW ( timeunit , windowsize, [offsetsize] )
Hopping windowHOPPINGWINDOW ( timeunit , windowsize , hopsize, [offsetsize] )
Sliding windowSLIDINGWINDOW ( timeunit , windowsize )
Javascript UDF intergration
• Supports stateless, compute only functions
• No external connectivity
• Why• REGEX
• Math libraries
Machine learning integration
Geospatial Queries (In public preview)• Geospatial Functions
• types• CreatPoint
• CreatePolygon
• CreateLineString
• Functions• ST_Distance
• ST_WITHIN
• ST_OVERLAP
• ST_INTERSECTS
Deployment options
Azure portal .Net SDK PowerShell
Pricing
• Streaming unit• 1 streaming unit = 1MB/Second
• 1 x 60 x 60 x 24 = 84.375 GB/day
• Data volume + streaming unit cost• Data volume = $ 0.001 or R 0.014 per GB
• Streaming unit = $ 0.12 or R1.656 per hour
• If run at max through put for a month• Total data ingested = 2.47 TB
• Data = R 35.44
• Streaming Unit = R1 192.32
• Total cost = R 1 227.76
DEMO
Questions