Download - (SDD409) Amazon RDS for PostgreSQL Deep Dive | AWS re:Invent 2014

November 12, 2014 | Las Vegas, NV

Grant McAlister, Amazon Web Services

Greg Roberts, illumina

0

2000

4000

6000

8000

10000

12000

14000

16000

18000

both on fsync=0 sync commit=0 fsync=0 & sync commit=0

Tran

sact

ion

s p

er S

eco

nd

32 thread insert- fsync vs sync commit

16 segments 256 segments

0

20

40

60

80

100

120

140

160

both on fsync=0 sync commit=0 fsync=0 & sync commit=0

Tim

e -

Seco

nd

s

Bulk load 2GB of data -fsync vs sync commit

16 segments 256 segments

29.1 28.8

26.125.223.9

0

5

10

15

20

25

30

35

fsync=1 & sync commit=0 fsync=0 & sync commit=0

Tim

e -

Min

ute

s

Index build on 20GB table

maintenance_work_mem=16MB &checkpoint_segments=16



29.1

23.4 23.9

28

0

5

10

15

20

25

30

35

maintenance_work_mem

Tim

e -

Min

ute

s

Index build on 20GB table

16

512

1024

4096

Table

Foo

Trigger

Table

Foo

Trigger

insert

Sync

Replication

Async Replication

AZ1 AZ2 AZ3

Reads10% Reads10% Reads10% Reads10%Reads10% Reads10% Reads10% Reads10%

Reads10%Writes10%

Reads10%Reads10%Writes10%



Reads10%

Reads10%Writes10%




Reads10%

Reads10%Writes10%Reads10%Writes10%Reads10%

Reads

90%

Writes10%

Primary

Writes10%

Replica1

Writes10%

Replica2

Writes10%

Replica3

Writes10%

Replica4

1X2X3X SCALE

1

2

3

4

5

6

7

8

1 2 4 8 16 32

Scale

Number of Nodes

Scale based on % Write

50% 40% 30% 20% 10%

AZ1 AZ2 AZ3

xlog1xlog2

xlog3xlog99

xlog1

xlog1

A - Foo

A- Bar

Source

A - Foo

A- Bar

Replica

0

1000

2000

3000

4000

5000

6000

7000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Tra

ns

ac

tio

ns

pe

r S

ec

on

d (

TP

S)

Hours

100% Read - 20GB data

db.m1.medium + 200GB standard

$0.575 per hour

0

1000

2000

3000

4000

5000

6000

7000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Tra

ns

ac

tio

ns

pe

r S

ec

on

d (

TP

S)

Hours



db.m3.medium + 200G + 2000 IOPS

$0.575 per hour

$0.408 per hour

0

1000

2000

3000

4000

5000

6000

7000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Tra

ns

ac

tio

ns

pe

r S

ec

on

d (

TP

S)

Hours




db.m3.large + 200G + 2000 IOPS

$0.575 per hour

$0.408 per hour

$0.508 per hour

0

1000

2000

3000

4000

5000

6000

7000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Tra

ns

ac

tio

ns

pe

r S

ec

on

d (

TP

S)

Hours





db.t2.medium + 200GB gp2

$0.105 per hour

$0.575 per hour

$0.408 per hour

$0.508 per hour

0

1000

2000

3000

4000

5000

6000

7000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Tra

ns

ac

tio

ns

pe

r S

ec

on

d (

TP

S)

Hours





db.t2.medium + 200GB gp2

db.t2.medium + 1TB gp2

$0.105 per hour

$0.575 per hour

$0.233 per hour

$0.408 per hour

$0.508 per hour

Custom docker job scheduler

– Simple design

– needed fast and reliable ACID transactions

– Semi-structured and relational data

API service

SpaceDock

Docker

BSFS - FUSE

Docker registry

Learn more at: developer.basespace.illumina.com

Mission control

mission.basespace.illumina.com

Elastic Load

Balancing

API Server(s)

on EC2

Mission Control Service

Amazon RDS

PostgreSQL

Amazon RDS

DB

(Multi-AZ)

Locked down Security Group

On-Demand Linux

EC2 instances with

mounted volumes

Schedules

Job

Native App AMI

BSFS -

FUSE drive

SpaceDock

AppInp

ut

Output

App Docker Image

Cancels

on failure

Amazon S3

Data Transfer

Developer Machines/

External Instances

Can Receive Jobs just

like on-demand

instance

Amazon Route 53

DNS

https://developer.basespace.illumina.com

http://mission.basespace.illumina.com

http://pgtune.leopard.in.ua/

Config Info Default New

Max_Connections Is based on memory and not CPU. Generally lower

this to match max settings of app pool.

InstanceMemory/125

82880 =2836

1000

Shared_Buffers Some may consider conservative, but depends on

app. 25% - 40% is ideal in 9.3.

InstanceMemory/327

68=8 GB

Same

Work_Mem Too conservative especially if you lower

max_connections

Engine Default= 1MB 3MB

Maintenance_Work_Mem Red flag, way too low. This determines vacuum and

index performance. Recommend ~1/16 system

memory.

Engine Default =

16MB

1.9 GB

Effective_Cache_Size Recommended range is 50-75% system memory. InstanceMemory/163

84 = 17 GB

Same

http://pgtune.leopard.in.ua/

+50% 0

2000

4000

6000

8000

db.m2.2xlarge db.m3.2xlarge

Total Time - Seconds (smaller better)

http://bit.ly/awsevals

http://bit.ly/awsevals