Postgres Scaling Opportunities and Options

Postgres Scaling Opportunities

BRUCE MOMJIAN

December, 2014

Configuring Postgres for heavy workloads can take many forms.This talk explores available Postgres scaling options.Creative Commons Attribution License http://momjian.us/presentations

1 / 37

PostgreSQL the database…

◮ Open Source Object Relational DBMS since 1996

◮ Distributed under the PostgreSQL License

◮ Similar technical heritage as Oracle, SQL Server & DB2

◮ However, a strong adherence to standards (ANSI-SQL 2008)

◮ Highly extensible and adaptable design

◮ Languages, indexing, data types, etc.◮ E.g. PostGIS, JSONB, SQL/MED

◮ Extensive use throughout the world for applications andorganizations of all types

◮ Bundled into Red Hat Enterprise Linux, Ubuntu, CentOSand Amazon Linux

Postgres Scaling Opportunities 2 / 37

PostgreSQL the community…

◮ Independent community led by a Core Team of six

◮ Large, active and vibrant community

◮ www.postgresql.org

◮ Downloads, Mailing lists, Documentation

◮ Sponsors sampler:

◮ Google, Red Hat, VMWare, Skype, Salesforce, HP andEnterpriseDB

◮ http://www.postgresql.org/community/


www.postgresql.org

http://www.postgresql.org/community/

EnterpriseDB the company…

◮ The worldwide leader of Postgres based products andservices founded in 2004

◮ Customers include 50 of the Fortune 500 and 98 of theForbes Global 2000

◮ Enterprise offerings:

◮ PostgreSQL Support, Services and Training◮ Postgres Plus Advanced Server with Oracle Compatibility◮ Tools for Monitoring, Replication, HA, Backup & Recovery

Community

◮ Citizenship

◮ Contributor of key features: Materialized Views, JSON, &more

◮ Nine community members on staff


EnterpriseDB the company…


Scaling

Database scaling is the ability to increase database throughput byutilizing additional resources such as I/O, memory, CPU, oradditional computers.However, the high concurrency and write requirements ofdatabase servers make scaling a challenge. Sometimes scaling isonly possible with multiple sessions, while other options requiredata model adjustments or server configuration changes.


Outline

Postgres scaling opportunities:

1. Multi-session

2. Single-session

3. Multi-host


Vertical/Horizontal Scaling

��

��

Horizontal

Vertical

��

��

��

��


Examples

Vertical scaling examples:

◮ More and faster CPUs

◮ More memory

◮ More and faster storage

◮ Battery-backed cache (BBU)

Horizontal scaling involves adding servers.


Hardware Components

I/O

CPU

Memory


1. Multi-Session

Clients

Database

CPU

I/O

Memory


I/O Spreading Using Tablespaces

Clients

Database

CPU

Memory

Tablespaces

Requires tables & indexes to be spread across tablespacesTablespaces should be on different storage devices


I/O Spreading Using Hash Partitions

��

��

Clients

Database

CPU

Memory

Hash Partitions

Requires rows to be hashed by key; one partition per device


I/O Spreading Using RAID 0

��

��

Clients

Database

CPU

Memory

RAID 0

Auto-hashed by storage block number


Write Spreading Using WAL Relocation

Clients

Database

CPU

Memory

WAL Tables and Indexes

Separates WAL writes from table & index I/O


Read Reduction via Increased Memory

Clients

Database

CPU

I/O

Memory

Additional memory caching reduces read requirements


Scaling Connections Using a Pooler

Clients

Database

Connection Pooler

CPU

I/O

Memory

Fewer idle connections reduces resource usage


Multi-Session CPU Scaling

Clients

Database

I/O

Memory

CPU

Multiple sessions spread across available CPUs


2. Single-Session

Client

Database

CPU

I/O

Memory


Read Parallelism Using effective_io_concurrency

Client

Database

CPU

Memory

Table

Index

Used during bitmap heap scansAssumes table is hashed across multiple devices


I/O Scaling via Parallelism (not implemented)

Client

Database

CPU

Memory

I/O

Involves parallel index, heap, partition, and tablespace access


CPU Scaling via Parallelism (not implemented)Database

Client

I/O

Memory

CPU

Involves parallel sorts, joins, and function execution


Sort I/O Reduction Using Work_mem

Client

Database

CPU

I/O

Memory Sorts/Hashes

Reduces temporary result reads & writes


Logical Dump Parallelism

CPU

I/O

Memory

pg_dump

Dumps tables using concurrent database connections


Logical Restore Parallelism

CPU

I/O

Memory

pg_restore

Loads tables and creates indexes using concurrent databaseconnections


3. Multi-Host

��

��

��

��


Read Scaling Using Pgpool & Streaming Replication

SELECTINSERT, UPDATE,

DELETE to master

host

Slave SlaveMasterreplication

to any host

replicationstreaming

��

��

��

��

pgpool

A full copy of the data exists on every node.


CPU/Memory Scaling With AsynchronousMulti-Master

SQL Queries SQL Queries

Asynchronous

CPU

Memory

CPU

Memory

Tables and Indexes Tables and Indexes

A full copy of the data exists on every node; requires conflictresolution. The asynchronous delay allows write-load buffering.


Oracle Real Application Clusters (RAC)

Synchronization

SQL Queries SQL Queries

CPU

Memory

CPU

Memory

CPU

Memory

WALWAL1 WAL2Tables and Indexes

Tables and indexes on shared storage; inter-node synchronizationrequired for cache consistency


I/O Scaling with Sharding: Challenges

◮ Multi-host write queries require two-phase commit (exceptXC)

◮ Multi-host visibility snapshots are not supported (except XC)

◮ Sharding benefits are only possible with a shardableworkload

◮ Changing the sharding layout can cause downtime

◮ Additional hosts reduce reliability; additional standbyservers might be required


Application-Based Sharding

��

��

��

��

��

��

Applications

SQL Queries

Data Node Data Node Data Node

Applications send queries based on the sharding layout.


Sharding Using PL/Proxy

��

��

��

��

��

��

Function Calls

SQL Queries

PL/Proxy

Data Node Data Node Data Node

Requires rows to be hashed by key, supports parallel-node queryexecution


Sharding Using Postgres-XC

��

��

��

��

��

��

SQL Queries

XC Coordinator

SQL Queries

Data NodeData Node Data Node

Joins, sorts, aggregates

ManagerGlobal Transaction

Enables hashing of large tables, replication of others; supportsparallel-node consistent transactions and DDL


Scaling Using Foreign Data Wrappers

��

��

��

��

��

��

SQL Queries

Foreign Server Foreign Server Foreign Server

SQL Queries

PG FDW

No joins, sorts, aggregates

Requires rows to be hashed by key


Bulk Data Scaling Using Hadoop

��

��

��

��

��

��

Map/Reduce Jobs

Postgres Server

Hadoop

Postgres Server Postgres Server


Additional Resources…

◮ Postgres Downloads:

◮ www.enterprisedb.com/downloads

◮ Product and Services information:

◮ [email protected]


www.enterprisedb.com/downloads

[email protected]

Conclusion

http://momjian.us/presentations thttps://www.flickr.com/photos/blieusong/


Postgres Scaling Opportunities and Options

Technology

Transcript of Postgres Scaling Opportunities and Options