Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size...
Transcript of Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size...
![Page 1: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/1.jpg)
Ike Ellis
Survey of the Azure
Data Landscape
![Page 2: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/2.jpg)
Wintellect Core Services
ConsultingCustom software application development and architecture
Instructor Led TrainingMicrosoft’s #1 training vendor for over 14 years having trained
more than 50,000 Microsoft developers
On-Demand TrainingWorld class, subscription based online training
![Page 3: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/3.jpg)
Industry InfluencersWe wrote the book (over 30 of them)
![Page 4: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/4.jpg)
Some Microsoft Related Highlights
Gold Azure Partner
2016 IAMCP Gold Partner of the Year for the U.S. announced at WPC
CEO is the Microsoft Regional Director (RD) for Atlanta
DevOps competency partner
Multiple ALM Rangers
Software Development competency partner
Xamarin Premier Consulting Partner
Multiple Xamarin Certified Engineers
Chosen to teach the 2-day Xamarin University pre-con at Evolve 2016
Other: Visual Studio Integration Partner, Azure Circle Partner, ALM Inner
Circle Partner, MVP of the Year, and more…
![Page 5: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/5.jpg)
Agenda
• Azure Blob Storage• Azure Table Storage• Azure CosmosDB• Azure SQL Database• Azure SQL in a VM• Azure SQL Data Warehouse• Azure Data Lake• Lots of other things supported:
• Postgres, MySQL, MongoDB, Redis
![Page 6: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/6.jpg)
Topic Agenda
• What is it?
• How is it used?
• What are the competitors?
• DEMO!
![Page 7: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/7.jpg)
Azure Blob Storage
![Page 8: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/8.jpg)
Azure Blob Storage
• Blobs are files (PDFs, JPGs, DOCs, etc)
• Highly durable, massively scalable
• More than 40 trillion stored objects
• 3.5+ Million requests/second
• Exposed via REST APIs
• Use them in .NET, C++, Java, Node.JS, Android…
• AzCopy, PowerShell
![Page 9: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/9.jpg)
Blob Storage Fault Tolerance & Scalability
![Page 10: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/10.jpg)
What kind of blobs can I have?
• Share files with clients
• off-load static content from web servers (invoices,
contracts, resumes)
• Azure Websites – Platform as a Service – no files on a
webserver
• SQL BAK Files
• VM Hard Drives
![Page 11: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/11.jpg)
Competitors?
• On premise SANS and arrays
• Amazon S3 Blob Storage
![Page 12: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/12.jpg)
Azure Blob Storage Demo
![Page 13: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/13.jpg)
Azure Table Storage
• Much of it similar to Azure Blob Storage
• Same scalability & redundancy
• Affordable price
• Very, very fast
• NoSQL key value pair solution
• Quick data retrieval, little configuration
![Page 14: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/14.jpg)
![Page 15: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/15.jpg)
![Page 16: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/16.jpg)
Competitors
• Amazon DynamoDB Table Storage
![Page 17: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/17.jpg)
Azure Table Storage Demo
![Page 18: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/18.jpg)
JSON Document
Standard for passing data between a server and a web application
Replacement for XML
Hierarchical
Terse
Simple data types
![Page 19: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/19.jpg)
Modeling in CosmosDB
{
"id": "1",
"firstName": "Thomas",
"lastName": "Andersen",
"addresses": [
{
"line1": "100 Some Street",
"line2": "Unit 1",
"city": "Seattle",
"state": "WA",
"zip": 98012
}
],
"contactDetails": [
{"email: "[email protected]"},
{"phone": "+1 555 555-5555", "extension": 5555}
Reading is one operation
Writing is one operation
No assembly de-assembly
![Page 21: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/21.jpg)
• MongoDB
• Amazon DynamoDB
Competitors
![Page 22: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/22.jpg)
![Page 23: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/23.jpg)
Azure SQL Database
• Platform as a Service
• All data is backed up for you
• Point in time restore
• Can be geo-redundant
• Scalable both in performance and in data size
• Up to 1TB
• Not feature complete with SQL Server in a VM
![Page 24: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/24.jpg)
Database Replicas
![Page 25: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/25.jpg)
https://azure.microsoft.com/en-us/documentation/articles/sql-database-transact-sql-information/
Azure SQL Database Unsupported Features
![Page 26: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/26.jpg)
You can also make it scale up!
![Page 27: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/27.jpg)
Amazon RDS
Competitors
![Page 28: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/28.jpg)
Azure SQL Database Demo
![Page 29: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/29.jpg)
• You manage backups
• You create fault tolerant options
• You manage disk space
• You manage patching
• You don’t manage hardware failure
• You don’t manage purchasing hardware
• You don’t manage networking infrastructure
Azure SQL Server in a VM
![Page 30: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/30.jpg)
• Use Premium Storage.
• Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.
• Use a minimum of 2 P30 disks (1 for log files; 1 for data files and TempDB).
• Keep the storage accountand SQL Server VM in the same region.
• Disable Azure geo-redundant storage (geo-replication) on the storage account.
• Avoid using operating system or temporary disks for database storage or logging.
Performance Considerations
![Page 31: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/31.jpg)
• Back up to Azure Blob Storage
• Use Always on Availability Groups and Windows Failover Clustering Services (WFCS) for fault tolerance
• Can use mirroring or log shipping too
• Can also mix in on-premise
Backups & Fault Tolerance
![Page 32: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/32.jpg)
Amazon EC2 – VMs in the cloud
Competitors
![Page 33: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/33.jpg)
• Elastic Massively Parallel Processing System
• Use T-SQL to query across relational and non-relational data
• Up to petabyte volumes of data
• Scale compute separately from data
• When paused, you only pay for storage
• Deploys in seconds
Azure SQL Data Warehouse
![Page 34: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/34.jpg)
• Supports 32 concurrent queries
• Used for fanning out queries over multiple machines for processing/aggregation/analytics
• Performance becomes far more predictable than with just straight SQL Server
• Not used in OLTP environments
Azure SQL Data Warehouse
![Page 35: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/35.jpg)
• A unit of scale that determines how much hardware will give great performance
• Done in increments of 100 (mostly)
• How many DTUs?
• Start Small
• Monitor
• Change as needed. It’s instant.
What is a DTU (Data Warehouse Unit)?
ALTER DATABASE MySQLDW MODIFY (SERVICE_OBJECTIVE = 'DW1000') ;
![Page 36: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/36.jpg)
Two choices:
• Distribute data based on hashing values from a single column
• Good if clusters of tables will be joined and are related
• Distribute data evenly but randomly
• Fail-safe method
Partitioning Data
![Page 37: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/37.jpg)
![Page 38: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/38.jpg)
Non-supported data types
•geometry, use a varbinary type•geography, use a varbinary type•hierarchyid, CLR type not native•image, text, ntext when text based use varchar/nvarchar (smaller the better)•nvarchar(max), use varchar(4000) or smaller for better performance•numeric, use decimal•sql_variant, split column into several strongly typed columns•sysname, use nvarchar(128)•table, convert to temporary tables•timestamp, re-work code to use datetime2 and CURRENT_TIMESTAMP function. •varchar(max), use varchar(8000) or smaller for better performance•uniqueidentifier, use varbinary(8)•user defined types, convert back to their native types where possible•xml, use a varchar(8000) or smaller for better performance - split across columns if needed
![Page 39: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/39.jpg)
• primary keys
• foreign keys
• check constraints
• unique constraints
• unique indexes
• computed columns
• sparse columns
• user-defined types
• indexed views
• identities
• sequences
• triggers
• synonyms
Unsupported Features
![Page 40: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/40.jpg)
Amazon RedShift
Competitors
![Page 41: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/41.jpg)
Azure SQL Data Warehouse Demo
![Page 42: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/42.jpg)
HDFS for the cloud
Can use tools like Spark, Storm, Flume, Sqoop, Kafka, etc.
No fixed limits on account size or file size
Azure Data Lake
![Page 43: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/43.jpg)
• An enterprise wide repository of every type of data collected in a single place
• Prior to any formal definition of requirements or schema. Allows every type of data to be kept without discrimination Organizations can then use Hadoop or advanced analytics to find patterns of the data.
• Serve as a repository for lower cost data preparation prior to moving curated data into a data warehouse.
What is a generic data lake?
![Page 44: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/44.jpg)
• Azure Data Lake Store – Built on HDFS
• Azure Data Lake Analytics – Built on Yarn. Introduces U-SQL/C#
Products
45
![Page 45: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/45.jpg)
A lot of Hadoop implementations, but nothing really quite like it
Competitors
46
![Page 46: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/46.jpg)
• MongoDB
• PostGres
• Redis
• MySQL
• Oracle
More data options….
47
![Page 47: Survey of the Azure Data Landscape - Wintellect€¦ · •Use Premium Storage. • Use a VM size of DS3 or higher for SQL Enterprise edition and DS2 or higher for SQL Standard edition.](https://reader036.fdocuments.in/reader036/viewer/2022081607/5ec66c3fd9efa86407282435/html5/thumbnails/47.jpg)
Ike Ellis
Ike Ellis, MVP blog.ikeellis.com
Book: Developing Azure Solutions
Podcast Guest: Talk Python to Me – Dec 2015, June 2016
.NET Rocks – Sept 2015, Sept 2016
SDTIG – www.sdtig.com