WINDOWS AZURE STORAGE – ARCHITECTURE VIEW

Chaowlert Chaisrichalermpol

Senior Developer

Jetabroad

AGENDA

Blob

Table

Queue

Storage services

MYSELF

Chaowlert Chaisrichalermpol

Azure Storage experiences

Architect of Samsung Gift & AIS Privilege

Monthly Active Users > 400,000

Daily Active Users > 150,000

Experienced migrate from Sql Azure to Table Storage

BLOB

BLOB – OVERVIEW

Store files

Files Media files Logs, raw data (consider using Azure Hadoop to analyze) Backup Any!

Cost is insanely cheap! (low as 24$/mth for 1TB!) This price include fault tolerance & API 48$/mth for geo replication

BLOB – DEMO

Movie provider

BLOB – BEST PRACTICES

Performance Point to Blob directly to reduce web role load Gzip text file before put into blob Resume download by Accept-Ranges header

Scalability Use CDN Scalability target per blob is 64M/sec Upload blob simultaneously is faster

TABLE

TABLE – OVERVIEW

NoSql key-value store

NoSql = No join + no grouping + no order by + no paging + no index + no transaction + no view + no stored proc + no function + no constraint + no default + no trigger + no fancy data type + no backup restore + much more!!!

TABLE – COMPARE WITH SQL AZURE

Azure Table SQL Azure

Features Basic Full sql features

Performance 20,000 rows/secHigher for cross accounts

180 concurrent connectionsHigher for premium services

Scalability Automatic Sql Federation

Availability Geo redundant Sql Data Sync

Cost (150 GB)

20$ (include 120M storage transactions)

226$ (no premium services, no federation, no data sync)

Usage Insert intensive data All others

TABLE – DEMO

Table Load Test

loader.io20 web roles

Azure Table

SQL Azure

TABLE – PERFORMANCE

Azure Table Sql Azure

Insert (DateTime Key) 2,113 tx/sec 588 tx/sec

Insert (Guid key) MAX!! 595 tx/sec

Update to single item 58 tx/sec 74 tx/sec

Read single MAX!! MAX!!

Read 1,000 179 tx/sec 860 tx/sec

TABLE – BEST PRACTICES

Review PartitionKey design Batch: Insert, update, delete up to 100 entities Ordering: Data always sorted by PartitionKey and RowKey

Scalability Scalability target for single partition is 2,000 rows/sec Avoid prepend/append pattern on PartitionKey Avoid hotspot on PartitionKey

Performance Use Json over AtomPub (70% bandwidth reduction) Use SCL 3.0+ (higher performance + use Json as default)

QUEUE

QUEUE – OVERVIEW

Persistent job processing

Able to process up to 32 messages at a time Not for log computing

Message size is up to 64kb May have to keep data in blob

QUEUE – VS SERVICE BUS

Azure Queue Service Bus

Max Age 7 days Unlimited

Protocols REST REST, SBMP, AMQP

Queue access Polling Polling, Long polling, Push

Feature Simple queue Session, Transaction, Duplicate detect, Auto dead lettering

Cost 0.05$ / million tx 1$ / million messages

Usage Job Messaging

QUEUE – HOW IT WORKS

Queue

Web Role

1. Add message

Worker Role

Message

2. Get message & lease

3. Delete message

QUEUE – SCENARIOS

Workload leveling Image resizing Order processing

Workload distribution Email sending Sending push notification

Temporal decoupling Night report jobs

QUEUE – DEMO

Ticket sell 300,000 users in 1 minute! Only 3,000 tickets available

QUEUE – BEST PRACTICES

Scalability Scalability target per queue is 2,000 message /sec Consider use multiple queues for higher scalability

Workload scaling AutoScaling by using MessageCount on queue

Cost of queue is worker role 99.9% worker role, 0.1% storage If usage is low, use web role as worker Use AutoScaling

STORAGE SERVICES

STORAGE – AVAILABILITY

LRS ZRS GRS RA-GRS

Store 3 repl single faculty

3 repl on multiple faculties

6 repl on 2 regions (3 repl each)

6 repl on 2 regions (3 repl each)

Durability Disk/node failure

Zone failure (fire)

Regional disaster

Regional disaster

Services Blob, Table, Queue

Block Blob only

Blob, Table, Queue

Blob, Table, Queue

Failover to secondary region

- - Auto within 15 mins

Failover read access by app

Price (first TB)/GB/month

$0.024 $0.030 $0.048 $0.061

STORAGE – SCALABILITY

Understand single partition unit

Blob Single Blob = Account + Container + Blob name Single Blob scalability target is 60MB/sec

Table Single Table Partition = Account + Table name + Partition Key Scalability target is 2,000 entities (1kb) / sec

Queue Single queue = Account + Queue name Single queue scalability target is 2,000 small message

(1kb)/sec

STORAGE – SCALABILITY

Scalability Target per account Capacity: up to 200TB Transaction: 20,000 messages (1kb) / second Network

Local redundant: Ingress 10Gibps/ Egress 15Gibps Geo redundant: Ingress 5Gibps/ Egress 10Gibps

Scale beyond single storage account

User Location Account

Lee Hong Kong Storage East

John London Storage USA

Amm Thailand Storage SE

STORAGE – PERFORMANCE

Storage & Web Role should be in the same region

Client can access storage directly (Public or SAS)

Use latest SDK

Net setting ServicePointManager.UseNagleAlgorithm = false; ServicePointManager.Expect100Continue = false; ServicePointManager.DefaultConnectionLimit = 100; (Or

More)

STORAGE – SECURITY

Public access (blob only) Direct access CDN access

Shared Access Signature (SAS) For client applications (mobile, web) Restricted access by permission: list, read, add, update,

delete Restricted access at data level Invalidate when regenerate full access key

Full access key For trusted services

STORAGE – DEMO

Todo list app (Share Access Signature )

ClientWeb Role

1. Request token

Azure Table

2. Access Data

Q & A