Best Practices for Becoming an Exceptional Postgres DBA

© 2014 EDB All rights reserved. 1

DBA Best Practices – Becoming an Exceptional Postgres DBA Marc Linster | August 27, 2014

© 2014 EnterpriseDB Corporation. All rights reserved. 2

•  EnterpriseDB Overview •  Best Practice Expertise •  The Postgres DBA Role

−  Monitoring −  Maintenance −  Capacity Planning −  Configuration Tuning −  Deployment Planning

•  Professional Development •  Summary and Resources •  Q&A

Agenda

We will take one quick-poll during the webinar to gather group data on best practices of interest


Brief EDB Overview


POSTGRES innovation

ENTERPRISE reliability

24/7 support

Services & training

Enterprise-class features & tools

Indemnification

Product road-map

Control

Thousands of developers

Fast development

cycles

Low cost

No vendor lock-in

Advanced features

Enabling commercial adoption of Postgres


Postgres Plus Advanced Server Postgres Plus

Cloud Database

High Availability Performance Management

REMOTE DBA 24x7

SUPPORT PROFESSIONAL

SERVICES

TRAINING

EDB Serves All Your Postgres Needs

PostgreSQL

Security


EDB Customers EDB currently has over 2,500 total customers including 50 of the Fortune 500 and 98 of the Forbes Global 2000


Best Practice Expertise


•  Remote DBA Services

•  Architectural Health Checks

•  Postgres Support Tickets

•  Postgres Training

Hundreds of Postgres Instances Supported

When •  ~70% support calls are at

deployment, post-deployment •  Remote DBA and AHC are

post-deployment What •  How to perform DBA tasks •  Which product tools or

features to use


When and Where do Users Need Help?

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

POC Dev Deployment Maintenance

Tuning

Product Awareness

How to

Enhancements

Corruption

Bug


Where Do Users Need Help # Tickets Phase Category POC Dev Deployment Maintenance Grand Total Connectors 0% 2% 1% 0% 3%

Bug 0% 0% 0% 0% 1% How to 0% 1% 0% 0% 1% Product Awareness 0% 1% 0% 0% 1%

Database 10% 9% 17% 37% 73% Bug 1% 2% 1% 2% 6% Corruption 0% 0% 0% 2% 2% Enhancements 0% 0% 0% 0% 1%

How to 4% 3% 6% 10% 23% Product Awareness 4% 3% 8% 15% 31% Tuning 0% 0% 1% 7% 9%

Replication 1% 4% 4% 2% 12% Bug 0% 1% 0% 0% 2% How to 0% 0% 3% 1% 4% Product Awareness 1% 2% 1% 0% 4% Tuning 0% 0% 0% 1% 1%

Utilities 1% 3% 6% 3% 12% Bug 0% 1% 1% 0% 2% Enhancements 0% 0% 0% 0% 0% How to 1% 1% 3% 1% 6% Product Awareness 0% 1% 3% 1% 5%

Grand Total 12% 17% 28% 42% 100%


Dep

loym

ent/D

atab

ase/

Pro

duct

Aw

aren

ess

& H

ow T

o

• How to set up connection pooling?

• How to set up hot standby and streaming replica?

• How to add table spaces? • How to partition? • How to upgrade from major

version? • What should I monitor? • How do I reduce downtime? M

aint

enan

ce/D

atab

ase/

Pro

duct

Aw

aren

ess

& H

ow T

o • How to find bloat in tables & indexes?

• How to enable auditing selectively without performance impact?

• What are the effects of changes to the values of postgresql.conf file parameters wal_writer_delay, bgwriter_delay?

• What is the correct configuration of hba_conf?

Select Top Support and RDBA Questions (not in rank order)


Monitoring

Maintenance

Performance Tuning

Capacity Management

Security

Deployment

Training & Professional Development

Building Blocks of the DBA Role


The Postgres DBA Role


•  Postgres is very reliable and stable

•  You can translate your existing DBA skills to Postgres

•  Key areas for Postgres DBAs −  Bloat management −  Vacuum setup −  OS configuration −  Storage sub-system configuration −  Backup – logical and physical −  HA Setup −  pg_hba.conf and other security settings −  Monitoring

Taking on the Postgres DBA Role


•  Begin with assessing the health of the DB - Architectural Health Check (AHC) −  Performance −  Maintenance Tasks – index management, bloat reduction −  Availability – backup strategies, recovery, replication and

failover −  Scalability and support for business and data growth −  Upgrade planning

•  Establish monitoring •  Focus on list of items uncovered in health check

Postgres DBA Role – An Approach


•  Technical architecture – OS, application, database

•  Indexes and bloat

•  Configuration tuning – OS, database −  Memory −  Write-ahead log −  Background worker −  Query −  Logging −  Checkpoint −  Vacuum

−  Query performance −  Security

Architectural Health Check Process

•  DR plan −  Backup strategy – logical and physical −  HA solutions – warm standby/log

shipping, hot standby, streaming and cascading replication

•  Performance tuning −  Reindexing −  Partitioning

•  Resource management −  Connections −  CPU

•  Hardware and OS configuration −  File system options


Monitoring and Maintenance


Maintenance Highlights from 20+ AHCs

Unused Indexes

39%

Bloated Indexes

30%

Bloated Tables 13%

Missing Indexes

9% Missing

PK 9%

Bloat management (43%) is a key

production problem

specific to Postgres


General Monitoring Recommendations Description Frequency Alerting Criteria /

Parameters monitored Load average, disk space and checking for .ready files

Every 10 minutes

(i) When load average is above threshold (ii) When disk space is below threshold (iii) When .ready(pg_xlog/archive_status/*.ready) files count is beyond threshold

Postgres instance is running and connection count

Every minute (i) When postgres instance is not running (ii) When number of connection is reached to value of max_connections (iii) When number of connections crosses threshold

Long running and waiting queries in database

Every 10 minutes

(i) When any query is running for longer than threshold (ii) When any query is in state of "idle in transaction" for more than threshold (iii) When any query is waiting for more than threshold

Bloats for tables and indexes

Once a day (i) When database tables and/or indexes individually bloated more than critical threshold (ii) When transaction ID wrap around is approaching

Is connection pooler running

Every 10 minutes

If pgbouncer or pgPool process is not running

Is streaming replication in sync with primary

Every 5 minutes

(i) When lag detected on streaming cluster exceeds threshold


•  Open source plug-in check_postgres.pl!−  http://bucardo.org/

•  PostgreSQL Statistics Collector - pg_stat_xxx views −  pg_stat_activity - details of open connections and running

transactions −  pg_stat_database - details of databases −  pg_stat_user_* - details of tables, indexes and functions −  pg_locks - list of current locks being held

•  Database object size functions −  pg_*_size - disk space used by a tablespace, database,

relation or total_relation (includes indexes and toasted data)

Monitoring Tools (if you can’t use PEM)

© 2014 EnterpriseDB Corporation. All rights reserved. 21 © 2011 EnterpriseDB . All rights reserved.

Postgres Enterprise Manager

•  Postgres Enterprise Manager (PEM) Tool −  Monitor real-time alerts −  Historical reports −  Identify poorly running SQL statements

•  Top Features −  Capacity Manager −  Audit Manager −  Postgres Expert −  SQL Profiler −  Alert Manager −  Log Manager −  Web Client −  Graphical Dashboards for I/O, Memory & user activity

© 2014 EnterpriseDB Corporation. All rights reserved. 22 © 2011 EnterpriseDB . All rights reserved.

PEM Features

•  Global dashboards with up-to-date status (up/down/performance) of all your servers

•  Ability to create performance thresholds for each key metric e.g. memory, storage, etc.

•  Any threshold violation results in an alert being sent to a centralized dashboard

•  All key performance-related statistics are automatically collected and retained

•  Forecast resource usage in the future

•  Identify and tune poorly running SQL statements

•  SNMP integration

•  Audit Logging for Postgres Plus Advanced Server instances


Capacity Management


•  Use cases: −  Consumption projection:

−  At current consumption rates when will I drop below 100MB of disk? −  At the consumption rate of the past 3 days when will I drop below 1 GB of

disk space? −  Point in Time Activity

−  Users were complaining of performance problems from 3 to 4 PM today. What was going on in the system, the database, how many users were connected?

−  We get a load spike every day at 11. What was happing at the time of the load spike?

−  Root cause for slower Query Performance −  Queries that access the employ table seem to be taking longer and longer.

View the growth of the number of dead tuples in the table, see what kind of scans have been executing against the table (index or sequential)

−  Activity monitoring −  Which database on the server is the most active? What kind of growth is it

seeing?

•  With PEM, you can: −  Evaluate past performance −  View trends with thresholds on certain values −  Plot metrics

Capacity Management


Sample PEM Report – Disk Space Available

25 © 2011 EnterpriseDB . All rights reserved.

Extrapolated Data Actual Data


Security


•  Conceptual Layers −  Authentication −  Authorization −  Auditing − Data Security −  SQL Injection Attacks

Security Auditing

•  Approach −  Secure physical access − Network access

limitation − Host access limitation − Database access

limitation − Data access limitation


Security Best Practices for PostgreSQL Security Best Practices for PostgreSQL and Postgres Plus Advanced Server

Executive Summary ................................................................3

Introduction ….........................................................................4

Postgres Security Features within the Above Framework ......6

Authentication .........................................................6

Authorization ….......................................................7

Accounting/Auditing ................................................8

Data Security ….......................................................9

SQL Injection Attacks ............................................10

Postgres Plus Advanced Server Security Features ..............11

Further Reading and Useful Links ........................................12

About EnterpriseDB ..............................................................12

© 2013 EnterpriseDB Corporation. All rights reserved. EnterpriseDB and Postgres Plus are trademarks of EnterpriseDB Corporation. Other names may be trademarks of their respective owners. http://www.enterprisedb.com

Page: 2

Table of Contents

Disclaimer The following is intended as an outline of EnterpriseDB’s general product direction. It is intended for informational purposes only, and it should not be relied upon in making purchasing decisions. This information may not be incorporated into any contract. It is not a commitment or obligation on the part of EnterpriseDB to release, launch or deliver any updates, modifications, material, code or functional improvements and may change at EnterpriseDB’s sole discretion.

http://info.enterprisedb.com/rs/enterprisedb/images/Whitepapers_Security_BP_PostgreSQL_and_Postgres_Plus_AS.pdf


Quick Poll Database Priorities


•  Question: What is your current database priority?

•  Answer choices: a.  Resolving a current performance issue b.  Architecture – migration, capacity planning, upgrade c.  Preparing for production deployment d.  Monitoring for ongoing database health e.  Establishing routine maintenance practices

Audience Quick Poll #1


Parameter Tuning for Performance


Performance Improvement Recommendations in: •  Optimizing memory usage

−  Adjusting shared_buffers −  work_mem * max_connections

•  vm.dirty_bytes and vm.dirty_background_bytes •  Understanding the storage

−  effective_io_concurrency

•  Identifying problem queries - pgbadger and SQL Profiler

The following analysis is based on a set of 20 AHCs executed in 2013

Lessons From AHC and RDBA


0%#

10%#

20%#

30%#

40%#

50%#

60%#

70%#

80%#

90%#

100%#

effec0ve_cache_size#

shared_buffers#

bgwriter_lru_maxpages#

wal_buffers#

maintenance_work_mem#

bgwriter_lru_mul0plier#

work_mem#

random_page_cost#

checkpoint_segments#

cpu_tuple_cost#

checkpoint_comple0on_target#

checkpoint_0meout#

log_line_prefix#

log_lock_waits#

log_min_dura0on_statement#

autovacuum_vacuum_scale_factor#

checkpoint_w

arning#

autovacuum_vacuum_cost_delay#

autovacuum_vacuum_threshold#

autovacuum_nap0me#

log_autovacuum_min_dura0on#

Top$Database$Parameter$Adjustments$3$Ordered$by$Frequency$

Red bars: experts expected top parameters, including vacuum_cost_limit, log_line_prefix, log_min_duration_statement, log_checkpoints

Top Database Parameter Adjustments – Ordered by Frequency


Kernel Parameters

0

2

4

6

8

10

12

14

16


Issue: •  Performance issues - speed Root Causes •  Slow, memory-hungry queries,

improper indexes Results

Customer Case Study

Use Case: Supply Chain Logistics IT Provider SaaS Warehouse Management System

Change Areas •  Indexing improvements •  Query tuning or rewrites


Issue: •  Intermittent timeouts Root Cause •  Parameter configuration

•  Enabled −  vm.dirty_bytes −  vm.dirty_background_bytes −  vm.zone_reclaim_mode

•  Disabled −  vm.overcommit_memory −  vm.swappiness −  redhat_transparent_hugepage

Customer Case Study

Use Case: Global MLS Platform – Reporting System

Change Areas •  Kernel parameters •  Database memory parameters

•  Increased −  work_mem −  maintenance_work_mem

•  Reduced −  shared_buffers

•  wal_buffers 32MB •  wal_level “archive”


Deployment Planning


•  Why do you need backup and recovery? Backup and Recovery Strategies protect you in case of: −  Catastrophic Device Failure −  Site Failure −  Maintenance −  Operator Error −  Compliance −  Data Corruption

•  Multiple components: −  Logical backup provides granularity in objects (tables, table

spaces, databases) −  Physical backups provide granularity in time for PITR −  Cold/off-line backups −  Hot/on-line backup, with WAL archiving

Backup and Recovery Strategies


Backup and Standby

hot 47%

warm 23%

None 18%

cold 12%

Standby Approach Physical

Only 36%

Logical Only 29%

Physical and

Logical 14%

FS Snapshot

7%

None 14%

Backup Approach 43% of customers could not execute PITR (prior to AHC) – some had NO backup in place

30% of customers would have been slow or unable to recover from failure


•  Backup Strategy Framework −  Logical backup after structural changes or

major updates of reference/meta data −  Physical backup (daily…) −  WAL file archiving −  Combine Backup/Recovery and Streaming Replication −  Uses replica to offload hot backup and logical backup

•  Consider −  Allowable PITR timeframe −  Data retention policy −  Test, test, test −  Periodic backup validation

•  Advanced solutions, such as RHCS, Veritas Clustering , EFM and EDB BART augment the basic capabilities

Backup and Recovery Best Practices


Stage: post-production Primary Issue: •  Outage due to replication

failure - insufficient log info to determine true root cause

Potential Pain Points •  Concerns have reached limits

of scale and meeting anticipated growth demands

Recommendations •  Backup and recovery strategies

−  Implement continuous archive recovery in addition to streaming replication

−  Revised backup strategy with PITR

−  Tune auto-vacuum settings and manually vacuum large tables

•  Indexes – create concurrent indexes, REINDEX to reduce bloat

•  Parameter tuning – various for kernel, memory settings and WALs

Customer Case Study

Use Case: on-line advertising service with hosted chat


•  Types of Storage Systems −  Direct Attached Storage (DAS) −  Storage Area Network (SAN) w. Fiber Channel −  Storage Area Network (SAN) w. iSCSI −  Network File System (NFS)

•  DAS: Fast and low latency •  SAN: Fast, expensive (w. Fiber Channel), scalable, can

include redundancy and smart file system operations •  NFS: Not an optimal solution for PostgreSQL data files

File System and Storage Subsystem


Summary and Useful Resources


•  Postgres DBAs need to focus on −  Config file settings, especially understanding memory configs −  Storage subsystem configuration −  OS Configuration −  Bloat & vacuum −  Backup - logical and physical −  HA setup −  Monitoring - see the Freight Train before it hits you!

•  Tools, tools, tools – Get the right tools

•  Most problems occur when moving to production −  Get ready early −  Get trained before MTP −  New Postgres DBA -- consider RDBA to help you get started

Summary


•  Training & Certification −  Classroom Comprehensive Admin - Bangalore – Sept 15-19 −  Comprehensive Admin Live Online – NA- with Bruce Momjian – PostgreSQL Guru

and community leader – Oct 6-10 −  Live Online EMEA – Adv. PostgreSQL Admin – with 9.4 Highlights – Oct 13-15

•  Get the team (developer, DBA, Infrastructure) ready to thrive with Postgres

•  EDB Knowledge Base

•  Blogs −  Robert Haas – EDB Chief Architect Database Server (http://rhaas.blogspot.com/) −  Vibhor Kumar – EDB Database Consultant (http://vibhorkumar.wordpress.com/)

•  User groups and Meetups – almost all major cities −  http://www.postgresql.org/community/user-groups/ −  http://postgresql.meetup.com/

•  PostgresOpen 2014 - Sept 17-19 Chicago

•  PostgreSQL Conference Europe 2014 - Oct 21-24 Madrid

What You Can Do


Architectural Health Check

•  Review of database, OS, env. settings

•  Guidance on tuning, backup, high availability strategies

Remote DBA Services

•  24 x 7 monitoring

•  Supplemental staff

•  Architectural health check

•  Access to Postgres expertise

Kick Start

•  Training •  Flex

consulting •  Developer

subscription

Training and Certification

•  Flexible formats: on-demand, live virtual, on-site

•  Continuing education and Postgres certification

How We Can Help

Whitepapers and Recorded Webinars http://www.enterprisedb.com/resources-community


For all attendees we are providing a special offer on related DBA services including training.

Please complete an inquiry form on-line and mention this webinar, or speak to your account manager

−  EDB’s regional sales offices found on enterprisedb.com: −  US +1 781-357-3390 or 1-877-377-4352 −  EMEA +31 (0) 70 36 11 774 −  JAPAN +81-50-5532-7038 −  KOREA +82-2-6007-2500 −  PAKISTAN +92-51-8358874 −  INDIA +91-20-30589500/01

−  Send email to [email protected]

−  http://www.enterprisedb.com/general-inquiry-form

Special Offers for Attendees


Q&A

Best Practices for Becoming an Exceptional Postgres DBA

Career

Transcript of Best Practices for Becoming an Exceptional Postgres DBA