Amazon RDS PostgreSQL: Enabling Innovation with Cloud Managed Databases
(SDD409) Amazon RDS for PostgreSQL Deep Dive | AWS re:Invent 2014
40
November 12, 2014 | Las Vegas, NV Grant McAlister, Amazon Web Services Greg Roberts, illumina
-
Upload
amazon-web-services -
Category
Technology
-
view
1.207 -
download
0
description
Learn the specifics of Amazon RDS for PostgreSQL's capabilities and extensions that make it powerful. This session covers database data import, performance tuning and monitoring, troubleshooting, security, and leveraging open source solutions with RDS. Throughout, this session focuses on capabilities particular to RDS for PostgreSQL.
Transcript of (SDD409) Amazon RDS for PostgreSQL Deep Dive | AWS re:Invent 2014
November 12, 2014 | Las Vegas, NV
Grant McAlister, Amazon Web Services
Greg Roberts, illumina
0
2000
4000
6000
8000
10000
12000
14000
16000
18000
both on fsync=0 sync commit=0 fsync=0 & sync commit=0
Tran
sact
ion
s p
er S
eco
nd
32 thread insert- fsync vs sync commit
16 segments 256 segments
0
20
40
60
80
100
120
140
160
both on fsync=0 sync commit=0 fsync=0 & sync commit=0
Tim
e -
Seco
nd
s
Bulk load 2GB of data -fsync vs sync commit
16 segments 256 segments
29.1 28.8
26.125.223.9
0
5
10
15
20
25
30
35
fsync=1 & sync commit=0 fsync=0 & sync commit=0
Tim
e -
Min
ute
s
Index build on 20GB table
maintenance_work_mem=16MB &checkpoint_segments=16
maintenance_work_mem=1024MB &checkpoint_segments=16
maintenance_work_mem=1024MB &checkpoint_segments=1024
29.1
23.4 23.9
28
0
5
10
15
20
25
30
35
maintenance_work_mem
Tim
e -
Min
ute
s
Index build on 20GB table
16
512
1024
4096
Table
Foo
Trigger
Table
Foo
Trigger
insert
Sync
Replication
Async Replication
AZ1 AZ2 AZ3
Reads10% Reads10% Reads10% Reads10%Reads10% Reads10% Reads10% Reads10%
Reads10%Writes10%
Reads10%Reads10%Writes10%
Reads10%Reads10%Writes10%
Reads10%Reads10%Writes10%
Reads10%
Reads10%Writes10%
Reads10%Reads10%Writes10%
Reads10%Reads10%Writes10%
Reads10%Reads10%Writes10%
Reads10%
Reads10%Writes10%Reads10%Writes10%Reads10%
Reads
90%
Writes10%
Primary
Writes10%
Replica1
Writes10%
Replica2
Writes10%
Replica3
Writes10%
Replica4
1X2X3X SCALE
1
2
3
4
5
6
7
8
1 2 4 8 16 32
Scale
Number of Nodes
Scale based on % Write
50% 40% 30% 20% 10%
AZ1 AZ2 AZ3
xlog1xlog2
xlog3xlog99
xlog1
xlog1
A - Foo
A- Bar
Source
A - Foo
A- Bar
Replica
0
1000
2000
3000
4000
5000
6000
7000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Tra
ns
ac
tio
ns
pe
r S
ec
on
d (
TP
S)
Hours
100% Read - 20GB data
db.m1.medium + 200GB standard
$0.575 per hour
0
1000
2000
3000
4000
5000
6000
7000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Tra
ns
ac
tio
ns
pe
r S
ec
on
d (
TP
S)
Hours
100% Read - 20GB data
db.m1.medium + 200GB standard
db.m3.medium + 200G + 2000 IOPS
$0.575 per hour
$0.408 per hour
0
1000
2000
3000
4000
5000
6000
7000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Tra
ns
ac
tio
ns
pe
r S
ec
on
d (
TP
S)
Hours
100% Read - 20GB data
db.m1.medium + 200GB standard
db.m3.medium + 200G + 2000 IOPS
db.m3.large + 200G + 2000 IOPS
$0.575 per hour
$0.408 per hour
$0.508 per hour
0
1000
2000
3000
4000
5000
6000
7000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Tra
ns
ac
tio
ns
pe
r S
ec
on
d (
TP
S)
Hours
100% Read - 20GB data
db.m1.medium + 200GB standard
db.m3.medium + 200G + 2000 IOPS
db.m3.large + 200G + 2000 IOPS
db.t2.medium + 200GB gp2
$0.105 per hour
$0.575 per hour
$0.408 per hour
$0.508 per hour
0
1000
2000
3000
4000
5000
6000
7000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Tra
ns
ac
tio
ns
pe
r S
ec
on
d (
TP
S)
Hours
100% Read - 20GB data
db.m1.medium + 200GB standard
db.m3.medium + 200G + 2000 IOPS
db.m3.large + 200G + 2000 IOPS
db.t2.medium + 200GB gp2
db.t2.medium + 1TB gp2
$0.105 per hour
$0.575 per hour
$0.233 per hour
$0.408 per hour
$0.508 per hour
Custom docker job scheduler
– Simple design
– needed fast and reliable ACID transactions
– Semi-structured and relational data
API service
SpaceDock
Docker
BSFS - FUSE
Docker registry
Learn more at: developer.basespace.illumina.com
Mission control
mission.basespace.illumina.com
Elastic Load
Balancing
API Server(s)
on EC2
Mission Control Service
Amazon RDS
PostgreSQL
Amazon RDS
DB
(Multi-AZ)
Locked down Security Group
On-Demand Linux
EC2 instances with
mounted volumes
Schedules
Job
Native App AMI
BSFS -
FUSE drive
SpaceDock
AppInp
ut
Output
App Docker Image
Cancels
on failure
Amazon S3
Data Transfer
Developer Machines/
External Instances
Can Receive Jobs just
like on-demand
instance
Amazon Route 53
DNS
http://pgtune.leopard.in.ua/
Config Info Default New
Max_Connections Is based on memory and not CPU. Generally lower
this to match max settings of app pool.
InstanceMemory/125
82880 =2836
1000
Shared_Buffers Some may consider conservative, but depends on
app. 25% - 40% is ideal in 9.3.
InstanceMemory/327
68=8 GB
Same
Work_Mem Too conservative especially if you lower
max_connections
Engine Default= 1MB 3MB
Maintenance_Work_Mem Red flag, way too low. This determines vacuum and
index performance. Recommend ~1/16 system
memory.
Engine Default =
16MB
1.9 GB
Effective_Cache_Size Recommended range is 50-75% system memory. InstanceMemory/163
84 = 17 GB
Same
+50% 0
2000
4000
6000
8000
db.m2.2xlarge db.m3.2xlarge
Total Time - Seconds (smaller better)
