Jason Volpe

Senior Solutions Architect

Microsoft Platform Team

SQL Server Best Practices For Storageand Instance Types on AWS

Thursday, 04/09/20

Choosing The Right AWS Storage For

SQL Server Workloads

SQL Server Storage Requirements

Purpose Optimization Capacity

DB files (mdf) Optimized for random IO Medium, Large, Very Large

Log files (ldf) Optimized for sequential IO Medium, Large

TempDB Optimized for lowest latency IO Medium, Large

Backup files Optimized for lowest cost Large, Very Large

-Ideally, we want the highest capacity for the lowest cost

-Find a balance between IOPs and Throughput; increasing one can decrease the other

-The amount of RAM SQL Server has access to significantly impacts what storage is

required

-Latency is also a critical component for SQL Server performance

AWS Has a Variety of Storage Options That Benefit SQL Server

Amazon EC2 Instance Store (NVMe/SSD Ephemeral Volumes)

Amazon Elastic Block Store (Amazon EBS)

Amazon FSx for Windows File Server, Amazon Elastic File System (Amazon EFS)

Amazon Simple Storage Service (Amazon S3)

Amazon Glacier

AWS Storage Gateway: File Gateway

AWS Snowball & AWS Snowball Edge

AWS Snowmobile

Amazon EBS (key capabilities for SQL Server database files)

• Persistent block level storage for EC2

• Pay only for what you provision

• Native redundancy and write cache

• Consistent and low-latency performance

• Optimized for random I/O

• Native support for encryption at rest

Amazon EBS Volume Types (Performance)

Volume TypeGeneral Purpose:

GP2Provisioned IOPS:

PIOPS/IO1Throughput Optimized:

ST1Cold HDD:

SC1

Technology: SSD SSD Magnetic Magnetic

Sizes: 1 GiB – 16 TiB 4 GiB – 16 TiB 500 GiB – 16TiB 500 GiB – 16 TiB

Max. IOPS: 16,000 @ 16KiB 64,000 @ 16KiB 500 @ 1MiB 250 @ 1MiB

Max. Throughput: 250 MiB/sec 1,000 MiB/sec 500 MiB/sec 250 MiB/sec

Max. IOPS / Instance: 80,000 80,000 80,000 80,000

Max. Throughput / Instance: 2,375 MiB/s 2,375 MiB/s 2,375 MiB/s 2,375 MiB/s

AWS Block

Storage

EC2 instance store

sc1st1

io1gp2

EBS

SSD-backed

volumes

EBS

HDD-backed

volumes

I3/I3en D2/H1

AWS Block Storage Options For SQL Server

Choosing an EBS Volume For Your SQL Server Workload

Throughput Heavy

Small, random I/O Large, sequential I/O

I3en

gp2 io1 sc1 st1

d2

Latency?

< 1 ms Single-digit ms ≤ 2,375 MiB/s

Aggregate throughput?

> 2,375 MiB/s

Which is more important?

Cost Performance

IOPS Heavy

≤ 80,000> 80,000

Which is more important?

Cost Performance

m5d r5

Provisioned IOPS

64k IOPs

1,000 mb/s

General Purpose

16k IOPs

250 mb/s

Throughput Optimized

500 IOPs

500 mb/s

Cold

250 IOPs

250 mb/s

Best Practice: RAID

When to RAID?

• Storage requirement > 16 TiB

• Throughput requirement > 1000 MiB/s

• Largest instance can support up to 2,375 MiB/s

• IOPS requirement > 64,000 @ 16K

• Largest instance can support up to 80,000 @ 16KEBS EBS EBS

Best Practice: RAID

Avoid RAID for redundancy

• Amazon EBS data is already replicated

• RAID1 halves available Amazon EBS bandwidth

• RAID5/6 loses 20% to 30% of usable I/O to parity

EBS EBS EBS

What’s An EC2 Instance Store?

Temporary block-level storage

Physically attached to host computer

Data lost when:Drive failure

Instance stops

Instance terminates

Instance store data loss prevention:SQL Server Always On AGs

Create RAID 1/5/6

Move data to Amazon S3 or EBS

Storage Pool using FSx or Storage Spaces Direct

Create a fault tolerant File System

EC2 Instances

Physical Host

Instance Store

or

Data persists on reboot!!

EC2 Instance Store Use Cases

I3en and R5d instance types

• Temporary storage (SQL Server tempdb, user dbs once hardened)

• Very low latency (0.1 ms)

• Extremely high IOPS (over 3 million vs. 80K for EBS-optimized)

• Higher throughput (almost 10x EBS max)

• Up to 60TB of NVMe SSD-based local attached storage

D2 instance type

• Up to 48 TB of HDD-based local attached storage

• High disk throughput

EC2 Instances

Physical Host

Instance Store

or

SQL Server Instant File Initialization

What Is Database File Initialization?• DB files are initialized by zeroing out leftover data on disk

• File initialization causes some DB operations to take longer

• Instant database file initialization reclaims unused disk space without zeroing it out

This excludes Log Files since they will always be zeroed when being created or growing in size.

SQL Server 2016+ Install Time

Enabling Instant Database File Initialization

Post-Install or Other Versions

Grant Perform volume maintenance tasks to SQL Server service account

1. Open the Local Security Policy app,

2. From Local Policy, choose User Rights Assignment.

3. Double-click Perform volume maintenance tasks.

4. Choose Add User or Group.

https://msdn.microsoft.com/en-us/library/ms175935.aspx

Storage Considerations + Sample Layout for SQL Server on EC2

IOPS and Throughput Considerations

• Enable EBS optimization on instance

• Create a volume for data (random access)

• Create a volume for log (sequential access)

• Format with 64K allocation unit size

• Stripe EBS PIOPS volumes for higher IOPS

• Match total EBS IOPS + throughput to instance type

Example volume layout:

C:\ Boot on General Purpose SSD (gp2)

D:\ Data files on PIOPS (single or striped)

E:\ Log file on st1 or PIOPS

F:\ Backups on st1, sc1, or S3

Z:\ Tempdb on instance storage (if available)

Consider Using multiple tempdb files (1:1 mapping with CPUs, up to 8)

tempdb on Instance Storage

Consider:

• Scripting/automating configuration on instance boot

• Changing SQL Server service startup to Automatic (Delayed Start) to allow instance storage to provision

• Striping multiple instance storage disks together for higher I/O

Choosing The Right EC2 Instance Type For

SQL Server Workloads

EC2 12+ years ago… A Single M1 For All Workloads

Scale up or down

quickly, as neededPay for what

you use “One size fits all for

all your SQL needs”

M1

© 2019, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.

EC2 Today… Over 175 Different Instances

Instance Family Included Instances

Compute Optimized c1, c3, c4, c5, c5d, c5n, cc2

FPGA f1

General Purpose a1, m1, m3, m4, m5, m5a, m5ad, m5d, m5dn, m5n, m6g, t2, t3, t3a

GPU g2, g3, g3s, g4dn, p2, p3, p3dn

ML ASIC inf1

Memory Optimizedcr1, m2, r3, r4, r5, r5a, r5ad, r5d, r5dn, r5n, u-12tb1, u-18tb1, u-24tb1,

u-6tb1, u-9tb1, x1, x1e, z1d

Micro t1

Storage Optimized d2, h1, hs1, i2, i3, i3en

© 2019, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.

Instance generation

r5.2xlargeInstance family Instance size

AWS Instance Cheat Sheet

* hs1.8xlarge has 17 vCPUs, c4.8xlarge has 36 vCPUs, d2.8xlarge has 36 vCPUs, regardless of being an 8xlarge

.large = 2 vCPU

.xlarge = 4 vCPU

.2xlarge x 4 = 8 vCPU

.3xlarge x 4 = 12 vCPU

.4xlarge x 4 = 16 vCPU

.6xlarge x 4 = 24 vCPU

.8xlarge x 4 = 32 vCPU

.9xlarge x 4 = 36 vCPU

.10xlarge x 4 = 40 vCPU

.12xlarge x 4 = 48 vCPU

.16xlarge x 4 = 64 vCPU

.18xlarge x 4 = 72 vCPU

.24xlarge x 4 = 96 vCPU

Multiply each .nxlarge instance size by 4

Instance type

SQL Server License Optimization with Optimize CPU

• Control active vCPUs and Hyper-Threading

status when launching new EC2 instances

• Reduce the number of SQL Server licenses

Instance

Type

Total

vCPUs

Active vCPUs with

Optimize CPUs

SQL Server

License Savings

r5.4xlarge 16 8 50%

r5.8xlarge 32 8 75%

SQL Server Instance Options – Balanced WorkloadsAccelerate SQL Server workloads that process large data sets in memory

SQL Server requirement AWS storage option

DB files (mdf) EBS: io1, gp2

Log files (ldf) EBS: st1

TempDB EC2 Instance Storage: SSD

Backup files st1, sc1, EBS snapshots

Features:-Up to 2,375 MB/s disk throughput

-Up to 80,000 IOPS

-Up to 3.9TB RAM

-Access to storage over dedicated network link

-Low latency using instance storage

-Instance types: m5, r5, m5d, m5a, m5ad, x1, x1e, z1d, …

Features:-Up to 16GB/s sequential disk throughput

-Up to 3,000,000 IOPS

-Up to 768GB RAM

-Supports 100 Gbps sustained network bandwidth

-Supports Elastic Fabric Adapter for low and consistent network latency

-Lowest latency using NVMe instance storage

-Instance types: I3, I3en, m5dn

• SQL Server Buffer Pool can leverage instance store NVMe disks safely for near zero latency caching

• Storage Pools can leverage instance store NVMe disks as a caching layer

• Use instance store NVMe disks for SQL Server online operations, such as maintenance

SQL Server Instance Options – Low Latency + High IOP WorkloadsDesigned for data-intensive workloads: Relational, NoSQL DBs, distributed file systems, search engines, and DW

SQL Server

requirement

AWS storage option

DB files (mdf) EC2 Instance Storage: NVMe/SSD/EBS Volumes

Log files (ldf) EC2 Instance Storage: NVMe/SSD/EBS Volumes

TempDB EC2 Instance Storage: NVMe/SSD

Backup files st1, sc1, NVMe to S3, EBS snapshots

© 2019, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.

EC2 Storage Optimized I3en InstancesDense NVMe SSD storage instances for data-intensive SQL Server workloads

Model vCPU Mem (GiB) Local Storage (GB) Network Bandwidth

i3en.large 2 16 1 x 1,250 NVMe SSD Up to 25 Gbps

i3en.xlarge 4 32 1 x 2,500 NVMe SSD Up to 25 Gbps

i3en.2xlarge 8 64 2 x 2,500 NVMe SSD Up to 25 Gbps

i3en.3xlarge 12 96 1 x 7,500 NVMe SSD Up to 25 Gbps

i3en.6xlarge 24 192 2 x 7,500 NVMe SSD 25 Gbps

i3en.12xlarge 48 384 4 x 7,500 NVMe SSD 50 Gbps

i3en.24xlarge 96 768 8 x 7,500 NVMe SSD 100 Gbps

Features:-Up to 60 TB of NVMe SSD instance storage Eliminate EBS costs by using NVMe that’s hardened

-Up to 16GB/s sequential disk throughput

-Up to 3,000,000 IOPS

-Up to 768GB RAM

-Up to 100 Gbps of network bandwidth using Elastic Network Adapter

-Up to 3.1 GHz Intel® Xeon® Scalable (Skylake) processors with new Intel Advanced Vector Extension (AXV-512)

instruction set

-Powered by the AWS Nitro System

EC2 Instance Type Purchasing Options For

SQL Server Workloads

Amazon EC2 Purchase Options For A New SQL Instance

Spot Instances*

Spare EC2 capacity at

savings of up to 90%

off On-Demand prices

Fault-tolerant, flexible,

stateless workloads

Reserved Instances

Make a 1 or 3-year commitment

and receive a significant discount off

On-Demand prices

Committed &

steady-state usage

On-Demand

Pay for compute capacity

the second with no

long-term commitments

Spiky workloads,

to define needs

* Spare EC2 capacity that AWS can

reclaim with a two-minute notice

What About Interruptions?

Minimal interruptions Over 95% of the instances were not

interrupted in the last 3 months

Check for 2-minute instance termination notice via instance

metadata or CloudWatch Events and automate by:

☑ Checkpointing

☑ Draining from ELB

☑ Using stop-start and hibernate to restart faster

The work you are doing to make

your applications fault-tolerant

also benefits Spot

Spot is optimized for stateless,

fault-tolerant, or flexible workloads.

Any application that can have part or

all of the work, paused and resumed or

restarted, can use Spot.

Leverage Spot Defined Duration To Guarantee Processes CompleteDefined Duration allows you to select the duration of hours you would like to reserve (1, 2, 3, 4, 5, or 6 Hours). Great for one-time daily batch processes like ETL, Staging

$0.27 $0.29$0.50

1b 1c1a

8XL

$0.30 $0.16$0.214XL

$0.07 $0.08$0.082XL

$0.05 $0.04$0.04XL

$0.01 $0.04$0.01L

C4

$1.76

On

Demand

$0.88

$0.44

$0.22

$0.11

EC2 Spot Pools - Instance Type Flexibility

Each Instance Family

Each Instance Size

Each Availability Zone (69)

In Every Region (22)

Is a Separate Spot Pool

R5M5

C5

I3 M5d

R4 D2C4

Questions Answers