Microsoft Azure StorageJai HaridasAndrew Edwards

DCIM-B384

IntroductionWhat’s New What’s ComingPatterns for Scalable & Resilient Apps

Agenda

Introduction

Microsoft Azure Storage• Cloud Storage - Anywhere and anytime

access• Blobs, Disks, Tables and Queues

• Highly Durable, Available and Massively Scalable • Easily build “internet scale” applications• More than 25 trillion stored objects• 2.5+ Million requests/sec on average

• Pay for what you use• Exposed via easy and open REST APIs,

Client Libraries and Tools

16 regions worldwide in 2014

Abstractions – Blobs and DisksBlobs – Massively scalable object store in the cloud

Simple REST interface (Put, Get, Delete)Data sharing – share documents, pictures, video, music, etc.Big Data – store raw data/logs and compute/map reduce over dataBackups – data and device backups

Disks – Network mounted durable disks for VMs in Azure

Move on-premises applications to cloudMounted disks are VHDs stored in Azure Blobs

Abstractions – Tables and QueuesTables – Massively scalable NoSQL cloud store

Key/Attribute(s) store at scaleAuto load balance partitions to meet traffic needsStore user, device or any type of metadata for your serviceOData protocol (AtomPub or JSON)

Queues – Reliable messaging system Reliable, low latency, high throughput messaging systemDecouple components/roles

Web role to worker role communicationAllows roles to scale independently

Implement scheduling of asynchronous tasksBuilding process/work flows

Additional Services, Tools and LibrariesAzure Import/Export • Move TBs of data into and out of Azure Blobs by shipping disks• Submit and monitor jobs via REST and Portal• All disks encrypted with BitLocker

Tools and Libraries• Client libraries• .NET, Java, C++, Node.js• Windows Phone & Windows Runtime• PowerShell commands• CLI tools• AzCopy – copy blobs and disks (tables later this year)• For backups, copying between accounts, and between on premise and

cloud

Geo Redundant Storage (GRS)Data geo-replicated across regions hundreds of miles apart

Provide data durability in face of potential major regional disastersProvided for Blob, Tables and Queues

User chooses primary region during account creationEach primary region has a predefined secondary region

Asynchronous geo-replicationOff critical path of live requestsUS West US East

US North US South

US Central US East 2

Europe North Europe West

Asia East Asia South East

China North China South

Japan East Japan West

South Brazil US South

Read-Only Access to GRS (RA-GRS) – Scenarios• Read-only access to secondary data even if

primary is unavailable• Access to an eventually consistent copy of the data in the other region

• For these, the application semantics need to allow for eventually consistent reads

• Recently GA’d

RA-GRS – How it Works? (1 of 2)• Customers using GRS can opt to have read-only

access to the eventually consistent copy of data on Secondary• Customer selects primary region, and the secondary region is fixed

• Get two endpoints for accessing your storage account• Primary endpoint• accountname.<service>.core.windows.net

• Secondary endpoint• accountname-secondary.<service>.core.windows.net

• Applications control which location they read data from• Use one of the above two endpoints• Our client libraries provides features to select location • Retry options: PrimaryOnly, SecondaryOnly, PrimaryThenSecondary, etc.

RA-GRS – How it Works? (2 of 2)• Same storage keys work for both

endpoints• Consistency

• All Writes go to the Primary• Reads to Primary are Strongly Consistent • Reads to Secondary are Eventually Consistent

• Applications can query the current max geo-replication delay for each service (blob, table, queue) in their storage account• Separate storage analytics metrics for

primary and secondary locations

RA-GRS – How it Works? (2 of 2)• Same storage keys work for both

endpoints• Consistency

• All Writes go to the Primary• Reads to Primary are Strongly Consistent • Reads to Secondary are Eventually Consistent

• Applications can query the current max geo-replication delay for each service (blob, table, queue) in their storage account• Separate storage analytics metrics for

primary and secondary locations

What’s new?

• Increased Scale Targets for Storage Accounts• Each storage account can hold up to 500TBs for all regions• Increased BW for US regions per storage account• 10Gbps Ingress and 20Gbps Egress

• Improved Versioning for Shared Access Signatures

• Client Libraries & Tools• .NET Library Desktop, Phone and Runtime with support for Files and Rest Version

2014-02-14• Java 1.0 RTM• Android 0.1 CTP • C++ Library CTP• AzCopy for Files CTP• PowerShell for Files CTP

• Azure Files Preview

What is New?

• Provide control to clients consuming SAS tokens (when sv=2014-02-14) to choose REST version using api-version query param

• Previously:• sv controlled the authentication, authorization and protocol version• sv is controlled by service distributing the SAS token• Clients may be a different version and not understand the protocol for version in sv

• Now:• Token Providers: Use 2014-02-14 version to create SAS tokens• Clients using storage client library: Just works• Clients building own REST protocol: Set api-version query param to intended version• Version for Authentication and authorization is still controlled by sv

Shared Access Signatures (SAS)

Use Storage Client Library 4.0 or later for generating SAS tokens

Clients using these tokens and storage client library will just work!Clients using these tokens to access storage resources but custom REST protocol implementation will need to add api-version query parameter

Use HTTPS when transmitting SAS tokensWhen resources like documents/html are accessed using SAS tokens via browsers, ensure content does not have clickable links

HTTP referrer header will expose SAS tokenUse javascript to block clicks or route it via custom application that remove SAS tokens

SAS Best Practices

Azure Files (Preview)

“I wish I could go to storage and provision a cloud drive, giving it a namespace, and that drive would then be UNC-addressable by the OSes.”

Azure Files – Customer Quotes

“I need two VM's running with a shared drive. One will write to the drive, the other will read [it].”

“Hi, I have two VM's in Microsoft Azure. All I want to do is set up a file share between them. Is this possible?”

“Is it possible to share a secondary disk between different VM instances?”

• Setup an IaaS VM to host a File Share backed by an IaaS Disk• Write code to find the IaaS File Share from the rest of the VMs in

your service.• Write some code to provide high availability • Handle host upgrades, node failures

• You can only access the File Share from other VMs

Sharing Files – The old way

IaaS VM

IaaS VM

IaaS VM

PaaS VM

IaaS VM(Sharing IaaS Disk)

Backup IaaS VMs (Mount/Share after failover)

Azure Files• Shared Network File Storage for Azure• Availability, durability, scalability are managed automatically• Supports two interfaces: SMB and REST

IaaS VM

IaaS VM

IaaS VM

PaaS VM

Azure File Share(PaaS)

• Share data across VMs and applications• Multiple writers, multiple readers using standard file system semantics.

• Share settings throughout services• VMs can read settings and files from a common, shared location.

These can be updated externally via REST.

• Dev/Test/Debug• Very useful to have a shared location for installing applications, setting

up VMs, running tools, and keeping notes while developing, testing, and debugging cloud services.

Azure Files - Scenarios

Azure Files - SMB 2.1 Protocol• Enables moving on-premises applications that

rely on shared file storage to Azure • Azure VMs can “net use” to a share

• Natively supported by OS APIs, libraries, and tools• Windows (CreateFile, ReadFile, WriteFile, …)• CRTs (fopen, fread, fwrite, …)• .Net (FileStream.Read, FileStream.Write, …)• Many more

• Supports standard file system semantics• Move and rename files and directories• Read-only, write through, overlapped• Change notifications

• Allows internet access to the same shared file system• Build hybrid applications (on premises +

cloud)• Supports a variety of common APIs:• Create/Delete Files and Directories• Write/Read Files• Get File and Directory properties• List Files

Azure Files - File REST APIs

Demo

Azure Files Part 1

Azure Files

Azure Files

Azure Files

Azure Storage Architecture

Massive Scale Out & Auto Load Balancing Index Layer

Distributed Replication Layer

Blob/DiskHead

QueueHead

TableHead

REST REST REST REST

File ShareHead

SMB

“Windows Azure Storage: A Highly Available Cloud Storage Service with Strong Consistency”, ACM Symposium on Operating System Principals (SOSP), Oct. 2011

Azure Files vs BlobsDescription Azure Blobs Azure Files

Durability Options

LRS, ZRS, GRS (and RA-GRS for higher availability)

LRS, GRS

Accessibility REST APIsSMB 2.1 (standard file system APIs)REST APIs

Connectivity REST – WorldwideSMB 2.1 - Within regionREST – Worldwide

Endpoints http://myaccount.blob.core.windows.net/mycontainer/myblob

\\myaccount.file.core.windows.net\myshare\myfile.txt

http://myaccount.file.core.windows.net/myshare/myfile.txt

DirectoriesFlat namespace however prefix listing can simulate virtual directories

True directory objects

Case Sensitivity of Names

Case sensitive Case insensitive, but case preserving

Capacity Up to 500TB containers 5TB file shares

Throughput Up to 60 MB/s per blob Up to 60 MB/s per share

Object size Up to 1 TB/blob Up to 1 TB/file

Billed capacity Based on bytes written Based on file size

Azure Files vs DisksDescription Disk Azure Files

Relationship with Azure VMs

Required for booting (OS Disk)

Scope Exclusive/Isolated to a single VM Shared access across multiple VMs

Snapshots and Copy Yes No

ConfigurationConfigured via portal/Management APIs and available at boot time

Connect after boot (via net use on windows)

Built-in authentication Built-in authentication Set up authentication on net use

Cleanup Resources can be cleaned up with VM if neededManually via standard file APIs or REST APIs

Access via RESTCan only access as fixed formatted VHD (single blob) via REST. Files stored in VHD cannot be accessed via REST.

Individual files stored in share are accessible via REST

Max Size 1TB Disk5TB File Share

1TB file within share

Max 8KB IOps 500 IOps 1000 IOps

Throughput Up to 60 MB/s per Disk Up to 60 MB/s per File Share

• Windows Supported:• Windows Server 2008 R2• Windows Server 2012• Windows Server 2012 R2

• Investigating Linux Support:• Ubuntu 13.10• Ubuntu 14.04 LTS

Azure Files – Client OS Support

• Request a token• Tokens will start to be granted in batches by end of May 2014

• Redeem token• Create new storage account• Create share (using powershell)• Put files into share (azcopy)• Connect to share from VM

Azure Files: Getting Started

Demo

Azure Files Part 2

Website Served From Azure File Share

Load Balancer

Azure VM

Azure VM

Azure File Share

…

REST APIs

SMB 2.1

Azure Files

Azure Files

What’s coming?

• Client Libraries• Node.js Library support for 2014-02-14 CTP• iOS Library CTP (By end of CY ‘14)• JavaScript Library CTP (By end of CY 14)

• Additional Durability Offering - Zone Redundant Storage for Block Blobs

What is Coming?

• LRS• Stores 3 replicas of the data within a single zone (facility) in a single region• Provides data durability for disk, node and rack failures

• ZRS• Available only for block blobs• Stores 3 replicas of the data across multiple zones (facilities).  Designed to keep all 3

replicas across zones within a single region, but may span across two regions. • Provides additional durability to protect data against zone failures (e.g., fire in a

facility)

• GRS• Stores 6 replicas of the data across two regions (3 in each region)• Provides additional durability to protect data against major regional disasters  (e.g., 

tornado, hurricane, earthquake, etc.)

3 Types of Durability offered for Azure Storage

Microsoft Azure Storage Pricing

Patterns for Scalable & Resilient Applications

• Scenario: Encode images that are being uploaded• Add a message to the queue with blob Uri

and maximum timeout to wait for blob upload• Upload the blob• Worker role processes messages• If blob not present, wait until the conservative max timeout• If blob exists, • Encode image and store encoded image• Delete the original image• Delete the message

Pattern for pre-processing resources

• Scenario: OneDrive needs to expand beyond a single account

• Create only as many accounts as needed today• Keep a map of user name+ bucket id to a storage

account name• When a bucket id fills up or account reaches limits

(capacity/throughput), create a new bucket id and pick a storage account from pool for storing data

Pattern – Scale beyond single storage account

User Name

Bucket Id

Storage Account

Storage Resource User Resource

brunopitman

1 jaidemo jaidemo/container/brunopitman/photo1.jpg

brunopitman/1/photo1.jpg

selmaramsey

1 jaidemo1 jaidemo1/container/selmaramsey/video1.wmv

selmaramsey/1/video1.wmv

brunopitman

2 jaidemo1 jaidemo1/container/brunopitman/backup.dat

brunopitman/2/backup.dat

selmaramsey

2 jaidemo22

jaidemo22/container/selmaramsey/resume.doc

selmaramsey/1/resume.doc

User Name

Bucket Id

Storage Account

Storage Resource User Resource

brunopitman

1 jaidemo jaidemo/container/brunopitman/photo1.jpg

brunopitman/1/photo1.jpg

selmaramsey

1 jaidemo1 jaidemo1/container/selmaramsey/video1.wmv

selmaramsey/1/video1.wmv

• Scenario: Table Queries for Tracking Game States• Web service maintains cache for game

state • On cache miss, retrieves state from Azure

Table• On failure, it fails fast and expects clients to

retry but asynchronously in the background retries • Retry on secondary and if secondary lag is

within X minutes use secondary data

Pattern for higher read availability & consistent low latency web requests

Demo

Read From Secondary

• Azure Storage• Durable, Scalable and highly Available Cloud Storage• Auto load balances to meet scale needs

• Azure Files – File shares in the cloud• Storage Durability Options – LRS, ZRS, and GRS• RA-GRS • Provides Higher Availability as applications can read from secondary when primary is

not available. • Client Library retries provides this capability out of the box

• Details on Internals can be found in the SOSP paper:• “Windows Azure Storage: A Highly Available Cloud Storage Service with Strong Consis

tency”, ACM Symposium on Operating System Principals (SOSP), Oct. 2011

Summary

Come Visit Us in the Microsoft Solutions Experience!

Look for Datacenter and Infrastructure ManagementTechExpo Level 1 Hall CD

For More InformationWindows Server 2012 R2http://technet.microsoft.com/en-US/evalcenter/dn205286

Windows Server

Microsoft Azure

Microsoft Azurehttp://azure.microsoft.com/en-us/

System Center

System Center 2012 R2http://technet.microsoft.com/en-US/evalcenter/dn205295

Azure PackAzure Packhttp://www.microsoft.com/en-us/server-cloud/products/windows-azure-pack

Resources

Learning

Microsoft Certification & Training Resources

www.microsoft.com/learning

msdn

Resources for Developers

http://microsoft.com/msdn

TechNet

Resources for IT Professionals

http://microsoft.com/technet

Sessions on Demand

http://channel9.msdn.com/Events/TechEd

Meet Storage Engineers

Storage engineering team will be available in the Azure booth area for the duration of Tech Expo (look for us under “Developer Platform & Tools”)You can also meet us at the “Ask The Experts/Meet and Geek” on Tuesday from 6:30-8:30pmWe look forward to meeting you!

Discuss your experience using Azure Storage

For customers already using Azure Storage in production workloads, we’d love to get your feedback at this Storage session:Date: Tuesday, May 13, 2014Time: 4p-5:30pLocation: Hilton Americas – Houston (connected to the convention center)Room # 3271600 Lamar Street, Houston

Complete an evaluation and enter to win!

Evaluate this session

Scan this QR code to evaluate this session.

© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.