#sqlsatParma
#sqlsat355November 22nd, 2014
Azure for SQL Server Developer
Marco Parenzan
#sqlsatParma
#sqlsat355November 22nd, 2014
Sponsors
#sqlsatParma
#sqlsat355November 22nd, 2014
Organizers
#sqlsatParma
#sqlsat355November 22nd, 2014
Speaker info/Marco Parenzan
Formazione & Divulgazione con 1nn0va
Microsoft MVP 2014 for Microsoft Azure
Formazione & Progettazione con ITS
Kennedy
Ricerca e Innovazione con Servizi Cgn
www.slideshare.net/marco.parenzan
www.github.com/marcoparenzan
marco [dot] parenzan [at] live [dot] it
www.innovazionefvg.net
@marco_parenzan
Cloud Architect
SQL Server Friend
Loves Functional Programming, Html5 Game Programming and Internet of Things
#sqlsatParma
#sqlsat355November 22nd, 2014
Agenda
Why Azure?
OPS Experience
DEV Experience
BI Experience
Conclusions
#sqlsatParma
#sqlsat355November 22nd, 2014
WHY AZURE?
#sqlsatParma
#sqlsat355November 22nd, 2014
Trends driving IT pressures & opportunities
#sqlsatParma
#sqlsat355November 22nd, 2014
Data storage
Cloud identity
VMs on demand
Disaster recovery
Deploying packaged applications
Moving existing applications to the public cloud
Utility ITExample scenarios
Utility ITStrategic
IT
#sqlsatParma
#sqlsat355November 22nd, 2014
The Microsoft data platform
#sqlsatParma
#sqlsat355November 22nd, 2014
SQL Server 2014 built-in hybrid technologies
#sqlsatParma
#sqlsat355November 22nd, 2014
Azure Data Services for Modern App Design
#sqlsatParma
#sqlsat355November 22nd, 2014
Barriers to Public Cloud AdoptionSecurity
Can a public cloud platform keep my data and applications safe??
ANSWER
You must learn to trust your
public cloud provider
#sqlsatParma
#sqlsat355November 22nd, 2014
Can I still meet my regulatory requirements in the public cloud??
Barriers to Public Cloud AdoptionCompliance
ANSWER
You must understand the rules
that apply to you
#sqlsatParma
#sqlsat355November 22nd, 2014
Barriers to Public Cloud AdoptionAvailability
Will public cloud datacenters be up when they’re needed??
ANSWER
They’ll be at least as good as
your own datacenter, and there
are SLAs
#sqlsatParma
#sqlsat355November 22nd, 2014
Barriers to Public Cloud AdoptionThe unspoken worry
Will public cloud platforms make me lose my job??
ANSWER
Probably not, but you’ll need to
learn some new skills
People who don’t like change shouldn’t work in the technology industry
#sqlsatParma
#sqlsat355November 22nd, 2014
OPS EXPERIENCE
#sqlsatParma
#sqlsat355November 22nd, 2014
Handle relational data with IaaS
You can create a VM with Windows Server
You can install any software you want if you have your licence
If you need a licence
In case you don’t have a licence, you have VM images ready with software and pay as you go (hour/minute basis)
You can scale in “minutes” …or in “real” hours…
…but not weeks (or at least days!)
#sqlsatParma
#sqlsat355November 22nd, 2014
Virtual Machine Storage Architecture
Azure Virtual Machine
C:\
OS DiskE:\, F:\, etc.
Data DisksD:\
Temporary Disk
(Contents can be lost)Disk Cache
#sqlsatParma
#sqlsat355November 22nd, 2014
The G family
Optimized for data workloads
Up to 32 CPU cores, 448 GB RAM,
6.5 TB local SSD
Latest generation Intel processor
G
#sqlsatParma
#sqlsat355November 22nd, 2014
Handle relational data with IaaS
Pro
Full power of a Relational DB
Simple Scaling Up
No licencing
Pay Per Use (no CAPEX)
Full power control
Ready images (no installation)
With specific configurations
Just Click, Name, choose,
deploy
Cons
OPEX: highest
Relational (non open) DBMS
are expensive
Bits+Management
You have to be a real
Administrator
Scalability: you can ONLY
scale up
Current sharding needs
application to be updated
Current sharding is in
experimentation stage
#sqlsatParma
#sqlsat355November 22nd, 2014
DEMO
#sqlsatParma
#sqlsat355November 22nd, 2014
AZURE STORAGE
One of the first two services (the other were Cloud Services)
#sqlsatParma
#sqlsat355November 22nd, 2014
Azure Storage Architecture
“Windows Azure Storage: A Highly Available Cloud Storage Service with Strong Consistency”, ACM
Symposium on Operating System Principals (SOSP), Oct. 2011
#sqlsatParma
#sqlsat355November 22nd, 2014
Durability Options
Locally Redundant Storage (LRS) Stores 3 replicas of the data within a single zone (facility) in a single region
Provides data durability for disk, node and rack failures
Geo Redundant Storage (GRS) Stores 6 replicas of the data across two regions (3 in each region)
Provides additional durability to protect data against major regional natural disasters
(e.g., tornado, hurricane, fire, etc, destroying a whole region)
Updates across regions are performed asynchronously
Zone Redundant Storage (ZRS) Stores 3 replicas of the data across multiple zones (facilities) within a single region or
across regions
Provides additional durability to protect data against zone failures (e.g., fire burning
down a facility)
#sqlsatParma
#sqlsat355November 22nd, 2014
Premium storage
Up to 32 TB of storage per VM
>50,000 IOPS per VM
Less than 1ms read latency
#sqlsatParma
#sqlsat355November 22nd, 2014
Abstractions – Tables
Tables – Massively scalable NoSQL cloud store Key/Attribute(s) store at scale
Store user, device or any type of metadata for your service
Auto load balances partitions to meet traffic needs
OData protocol (AtomPub or JSON)
#sqlsatParma
#sqlsat355November 22nd, 2014
Abstractions – Azure Storage Files
Move on-premises applications to cloud
VMs can net use an SMB share using standard file APIs and semantics
SMB 2.1 protocol
VM and storage account within same region
Supports REST and SMB protocol access to same file share
Azure Storage
Blobs
Tables
Queues
Files
Share data stored in Azure Files
among Azure VMs via SMB
Microsoft Azure
SMB
RES
T
API
RES
T
API
#sqlsatParma
#sqlsat355November 22nd, 2014
Abstractions – Blobs
Blobs – Massively scalable object store in the cloud Simple REST interface (Put, Get, Delete)
Data sharing – share documents, pictures, video, music, etc.
Big Data – store raw data/logs and compute/map reduce over data
Backups – data and device backups
#sqlsatParma
#sqlsat355November 22nd, 2014
Virtual Machine Storage Architecture
Azure Virtual Machine
C:\
OS DiskE:\, F:\, etc.
Data DisksD:\
Temporary DiskDisk Cache
G:\, H:\, etc.
SMB Share
#sqlsatParma
#sqlsat355November 22nd, 2014
Make a SQL backup over Azure
Backup works if move data in a far and safe place
You do it on a daily basis
It can be tested
SQL Server 2012/2014 can backup over Azure Storage Set azure credentials in SQL Server to authenticate on-
premise SQL over Azure (share a certificate between Azure and on-premise host)
Indicate the container and the blob you configure for this activity
Do the backup
Restore backup to the new host
#sqlsatParma
#sqlsat355November 22nd, 2014
Breaking News – Premium Storage
Consistent low latency SSD based with predictable IO throughput
Suitable for high-performance IO-intensive database workloads
Single digit milliseconds latencies
Supports up to 1 TB blob/disk size
Stripe up to 32 disks for a total of
32TB and more than 50,000 IOPS
Premium Storage Disks work in
conjunction with a new VM series
#sqlsatParma
#sqlsat355November 22nd, 2014
DEMO
#sqlsatParma
#sqlsat355November 22nd, 2014
Handle relational data with SaaS
You don’t want to handle management issues
You don’t want vertical scalability-related
issues
You don’t need special infrastructural
architecture
#sqlsatParma
#sqlsat355November 22nd, 2014
Microsoft SQL Platform Continuum
Physical
Virtual
Infrastructure as a Service
Platform as a Service
SQL Server
Physical Machines (raw iron)
SQL Server Private Cloud
Virtualized Machines + Appliances
SQL Server in Azure VM
Virtualized Machines
Azure SQL Database
Virtualized DatabasesSQL
Dedicated
Higher Cost
Shared
Lower
Cost
Higher Administration Lower Administration
Hybrid
Cloud
On Premises
Off Premises
SQL SQL
SQL SQL SQL
SQL SQL SQL
SQL
#sqlsatParma
#sqlsat355November 22nd, 2014
Azure SQL Database Service Tiers
• •• •••
• •• •••
#sqlsatParma
#sqlsat355November 22nd, 2014
Scalable Performance
Basic, Standard, and Premium provide increasing performance levels
Relative performance is expressed in database throughput units (DTUs)
DTU is a blended measure of CPU, read IO, write IO, and memory
Scale performance up/down via portal, APIs, PS, or T-SQL
to reflect actual or anticipated demand
Database remains online while scaling
Hourly billing improves cost efficiency
Performance is easily scaled up or down to
meet changing workload and business needs
B S0S1
S2P1
P2
P3
#sqlsatParma
#sqlsat355November 22nd, 2014
Data Protection
Automatic, continuous backups
Retention varies by service tier: Basic 7 days, Standard 14 days, Premium 35
days
Daily backups also copied to a remote datacenter
“Oops” recovery
Self-service point-in-time restore from local backups, including deleted
databases
Data center disaster
Self-service restore from remote backups to any region at any time
No charge for backups, normal database rates apply for restored databases
Basic, Standard & Premium databases are
protected from accidental data corruption, deletion
& disaster
#sqlsatParma
#sqlsat355November 22nd, 2014
Handle relational data with SaaS
Pro
OPEX: lower than IaaS
Bits+Management
Full power of a Relational DB
Editions
Clustering
CPU+Memory
No licencing (No CAPEX)
Pay Per Use (OPEX)
No need for images
Cons
OPEX: lower, than IaaS, but
more than other
Partial power of a Relational
DB
…
Configure what is just
available
Scalability: can ONLY scale up
#sqlsatParma
#sqlsat355November 22nd, 2014
Migrating your on-premise SQL to SQL
Azure
Not all features of On-Premise SQL are availavable in SQL Azure A “backup” is not enough to bring DB to the cloud
Exporting a script
Some tools are available, updated Developer Way: Database Project, targeting SQL
Azure
Operations Way: Database | Tasks | Deploy to Windows Azure SQL Database
https://sqlazuremw.codeplex.com/
#sqlsatParma
#sqlsat355November 22nd, 2014
SQL Azure performance considerations
Microsoft is making huge investments
Compatibility with full t-sql will improve
DB per customer
Hundreds/thousands of databases
#sqlsatParma
#sqlsat355November 22nd, 2014
DEMO
#sqlsatParma
#sqlsat355November 22nd, 2014
Lower costs and improve predictability
Automation
Enable service owners to focus on work that adds business value
Reduce error-prone manual activities while lowering costs
Process automation that simplifies cloud
management
Optimize and extend
existing investments
IntegrationIntegrate into existing systems with
PowerShell integration modules
Build additional PS modules to
enable integrating into other
systems
Deliver flexible and reliable services
Orchestration
Accelerate time to value with flexible process workflows
Improve service reliability across multiple tools, systems, and department silos
#sqlsatParma
#sqlsat355November 22nd, 2014
Built on PowerShell Workflow
PowerShell Workflow
•Use Windows PowerShell syntax
•Multi-device management
•Running a single task to manage complex, end-to-end processes
•Automated failure recovery
•Connection and activity retries
Centralized store
•Credentials / certificates
•Global variable
•Global connection for runbooks
•Modules
•Runbooks (draft / published versioning)
•Scheduling
Highly Available
•Runbook servers to process jobs
•Odata Web service to submit / retrieve status
•SQL Server clustering / always on
Historical Analysis
•Historical view of runbook jobs
•Reporting through Excel PowerPivot for ROI
•View runbook used for all jobs
PowerShell Workflow
Centralized store
Highly Available
Historical Analysis
Microsoft Azure
Automation
#sqlsatParma
#sqlsat355November 22nd, 2014
Runbook Authoring
Author Create PowerShell Workflow
runbooks
Call existing runbooks in library
Manage and browse and insert assets in runbooks Modules and Activities
Credentials (PowerShell Credentials and Certificates)
Variables (standard and encrypted)
Connections
Schedules
Test Run runbook and see results within
authoring window
Troubleshoot issues
Publish Edit draft before publishing
#sqlsatParma
#sqlsat355November 22nd, 2014
Manage Runbooks & Jobs
Dashboard View View runbook jobs over time
Find jobs that may need attention
Runbooks View Filtering of jobs based on status and
date
Identify authoring state (New, In edit, Published)
Filter by tags to group related runbooks
Jobs View History of jobs per runbook
Who last updated and when
Input parameters and output
Drill into each job to view streams generated to help troubleshooting
Stop, Suspend and Resume job
#sqlsatParma
#sqlsat355November 22nd, 2014
DEMO
#sqlsatParma
#sqlsat355November 22nd, 2014
Resource Manager
Today Challenge
Deploy or update a group of resources, repeatedly
Manage permissions on a group of resources
Visualize a group of resources in a logical view,
including monitoring/billing
Resource Manager
Application Lifecycle Container
Declarative solution for Deployment and
Configuration
Consistent Management Layer
#sqlsatParma
#sqlsat355November 22nd, 2014
Resource Groups
Tightly coupled containers of multiple resources of similar or different types
Every resource *must* exist in one and only one resource group
Resource groups can span regions
Resource Group is a unit of management
Lifecycle: deployment, update, delete, status
Grouping: metering, billing, quota: applied & rolled up to group
#sqlsatParma
#sqlsat355November 22nd, 2014
Instantiation of repeatable config.Configuration Resource Group
Power of Repeatability
Azure Templates can: Ensure Idempotency
Simplify Orchestration
Provide Cross-Resource Configuration and Update Support
Azure Templates are: Source file, checked-in
Specifies resources and dependencies (VMs, WebSites, DBs) and connections (config, LB sets)
Parametizedinput/output
SQL - A WebsiteVirtualMachines
SQL-AWebsite[SQL CONFIG] VM (2x)
DEPENDS ON SQLDEPENDS ON SQL
SQL CONFIG
#sqlsatParma
#sqlsat355November 22nd, 2014
DEMO
#sqlsatParma
#sqlsat355November 22nd, 2014
DEV EXPERIENCE
#sqlsatParma
#sqlsat355November 22nd, 2014
The big issue leading to CQ(R)S
A single model (tipicallyfrom a Database schema) caring about all aspects of the domain is hard
Complex aggregate roots that fits costly intoa relational db mapping cost is higher
than access cost (crud)
Eventual consistency is a must Acid costs
Presentation layer
Application layer
Infrastructure layer
Domain layer
#sqlsatParma
#sqlsat355November 22nd, 2014
Transaction Script
Data Storage
Application Services
Remote Facade
Client
Request DTO
DTO Returned
Send DTO
Status Returned
#sqlsatParma
#sqlsat355November 22nd, 2014
Active Record
Data Storage
Application Services
Remote Facade
Client
Request DTO
DTO Returned
Send DTO
Status Returned
Entity Entity
#sqlsatParma
#sqlsat355November 22nd, 2014
Command/Query Responsibility Segregation
Domain Model in the Command stack Domain Model focused on behavior that alters state
Ad-hoc storage for queries in the Query stack Table-per-view strategy
Query through LET
Command/Query Separation (cit. Bertrand Meyer – 1980s)
Query
> Returns data
> Doesn’t alter state
Command
> Alter state
> Doesn’t return data
#sqlsatParma
#sqlsat355November 22nd, 2014
CQRS in a sketch
Command Handlers
Client
Send Command
Ack/Nak
Response
Domai
n
Object
Domain
Object
Repository
Query Handlers
Data Storage
Event Bus
Eve
nt
Ha
nd
ler
s
Request DTO
DTO Returned
#sqlsatParma
#sqlsat355November 22nd, 2014
Write DB with DocumentDB
Produce and consume data at a staggering rate while requiring near instantaneous response times to match user expectations
Developed iteratively with many versions supported concurrently and continuously evolving data models
Increasingly complex, compensating for the lack of query and transactional processing in today’s NoSQL database systems
Experience unpredictable and explosive growth
It is a fully managed, highly scalable, queryable, truly schema-free JSON document database, delivered as a service for modern applications
SQL
query over schema-free JSON
multi-document
transactions
tunable, high performance
designed
cloud first
#sqlsatParma
#sqlsat355November 22nd, 2014
DEMO
#sqlsatParma
#sqlsat355November 22nd, 2014
The CAP Theorem
Theorem: You can
have at most two of
these properties for
any shared-data
systemConsistency
Partition
tolerance
Availability
#sqlsatParma
#sqlsat355November 22nd, 2014
Consistency
Two kinds of consistency:
strong consistency – ACID(Atomicity Consistency
Isolation Durability)
weak consistency – BASE(Basically Available
Soft-state Eventual consistency )
#sqlsatParma
#sqlsat355November 22nd, 2014
Eventual Consistency
When no updates occur for a long period of time, eventually all updates will propagate through the system and all the nodes will be consistent
For a given accepted update and a given node, eventually either the update reaches the node or the node is removed from service
Known as BASE (Basically Available, Soft state, Eventual consistency), as opposed to ACID
#sqlsatParma
#sqlsat355November 22nd, 2014
Availability
Traditionally, thought of as the server/process
available five 9’s (99.999 %).
However, for large node system, at almost
any point in time there’s a good chance that a
node is either down or there is a network
disruption among the nodes.
Want a system that is resilient in the face of
network disruption
#sqlsatParma
#sqlsat355November 22nd, 2014
Asynchronous Geo-replication with SQL
Azure
Standard geo-replication (Standard and Premium) Optional non-readable secondary in paired region
Secondary database is charged at discounted rate
Failover enabled by Microsoft if disaster occurs
Active geo-replication (Premium) Up to 4 readable secondaries
Full control over secondary location and failover
Supports load balancing, application upgrade and relocation scenarios
Can be combined with a non-readable secondary
Geo-replication minimizes business
interruption from possible disasters
DB
1DB
1
DB
1DB
1
DB
1
#sqlsatParma
#sqlsat355November 22nd, 2014
DEMO
#sqlsatParma
#sqlsat355November 22nd, 2014
Azure Caching
Opportunity
Lower accesses to SQL Azure and cost less
Cache the page?
Cache the content of a Page
#sqlsatParma
#sqlsat355November 22nd, 2014
Handle your “repeatable data” with Redis
Declarative approach Configure ASP.NET MVC to use Redis
Session State or Output cache
Look At Andrea Dottor presentazione about it
Redis as an efficient in memory key/value store
Find many .NET clients in nuget Example Stack Exchange implementation
Implements a cache-aside pattern
Query cache first If not found, Query datasource and storie info incache
Cache invalidaton rules have to be handled Explicit
Timeout
#sqlsatParma
#sqlsat355November 22nd, 2014
DEMO
#sqlsatParma
#sqlsat355November 22nd, 2014
Azure Search
A search-as-a-service solution allowing
developers to incorporate great search
experiences into applications without
managing infrastructure or needing to
become search experts.
It is based on Elastic Search
#sqlsatParma
#sqlsat355November 22nd, 2014
DEMO
#sqlsatParma
#sqlsat355November 22nd, 2014
New development challenges
None of the following is true:
“Network is reliable”
“Latency is near to zero or irrelevant”
“Bandwidth is unlimited”
“Network is secure”
“Topology doesn’t change”
“Transport cost is irrelevant”
“Network is homogeneous”From Mauro Servienti presentation
http://www.slideshare.net/mauroservienti/in-the-brain-designing-distributed-scalable-and-reliable-systems
#sqlsatParma
#sqlsat355November 22nd, 2014
Latency and Retries
SQL Server
Long latencies and timeouts
Transient errors
Retry strategy
#sqlsatParma
#sqlsat355November 22nd, 2014
DEMO
#sqlsatParma
#sqlsat355November 22nd, 2014
Q&A
Questions?
Email: [email protected]
Slideshare:
http://www.slideshare.net/marcoparenzan
GitHub:
https://www.gihtub.cob/marcoparenzan
Twitter: http://twitter.com/marco_parenzan
#sqlsatParma
#sqlsat355November 22nd, 2014
Thanks!
#sqlsatParma
#sqlsat355
Top Related