masterpgadministration-091023104603-phpapp01

7/28/2019 masterpgadministration-091023104603-phpapp01

1/111

MasteringPostgreSQL Administration

BRUCE MOMJIAN,

ENTERPRISEDB

April, 2009

AbstractPOSTGRESQL is an open-source, full-featured relational database.This presentation covers advanced administration topics.

Creative Commons Attribution License http://momjian.us/presentations


2/111

Introduction

Installation

Configuration

Maintenance

Monitoring

Recovery

Mastering PostgreSQL Administration 1


3/111

Installation

Click-Through installers

MS Windows

Linux OS/X Solaris

Ports

RPM DEB PKG other packages

Source

obtaining build options

installingMastering PostgreSQL Administration 2


4/111

Initialization (initdb)

$ initdb

The files belonging to this database system will be owned by user "postgres".

This user must also own the server process.

The database cluster will be initialized with locale C.

The default database encoding has accordingly been set to SQL_ASCII.The default text search configuration will be set to "english".

creating directory /u/pg/data ... ok

creating subdirectories ... ok

selecting default max_connections ... 100

selecting default shared_buffers/max_fsm_pages ... 32MB/204800

creating configuration files ... okcreating template1 database in /u/pg/data/base/1 ... ok

initializing pg_authid ... ok

initializing dependencies ... ok

creating system views ... ok

loading system objects descriptions ... ok

creating conversions ... okcreating dictionaries ... ok

setting privileges on built-in objects ... ok

creating information schema ... ok

vacuuming database template1 ... ok

copying template1 to template0 ... ok

copying template1 to postgres ... okMastering PostgreSQL Administration 3


5/111

Initialization (continued)

WARNING: enabling "trust" authentication for local connections

You can change this by editing pg_hba.conf or using the -A option the

next time you run initdb.

Success. You can now start the database server using:

postgres -D /u/pg/data

or

pg_ctl -D /u/pg/data -l logfile start



6/111

pg_controldata

pg_control version number: 74

Catalog version number: 200502281

Database system identifier: 4766833642862247929

Database cluster state: shut down

pg_control last modified: 03/03/05 10:49:18Current log file ID: 0

Next log file segment: 1

Latest checkpoint location: 0/A34010

Prior checkpoint location: 0/A2D5C0

Latest checkpoints REDO location: 0/A34010

Latest checkpoints UNDO location: 0/0Latest checkpoints TimeLineID: 1

Latest checkpoints NextXID: 545

Latest checkpoints NextOID: 17233

Time of latest checkpoint: 03/03/05 10:49:18

Database block size: 8192

Blocks per segment of large relation: 131072Bytes per WAL segment: 16777216

Maximum length of identifiers: 64

Maximum number of function arguments: 32

Date/time type storage: floating-point numbers

Maximum length of locale name: 128

LC_COLLATE: CLC_CTYPE: C



7/111

System Architecture

utility

Plan

Optimal Path

Query

Postmaster

Postgres Postgres

Libpq

Main

Generate Plan

Traffic Cop

Generate Paths

Execute Plan

e.g. CREATE TABLE, COPYSELECT, INSERT, UPDATE, DELETE

Rewrite Query

Parse Statement

UtilityCommand

Storage ManagersCatalogUtilities

Access Methods Nodes / Lists



8/111

Starting Postmaster

LOG: database system was shut down at 2009-01-27 10:32:05 EST

LOG: database system is ready to accept connectionsLOG: autovacuum launcher started

manually

pg_ctl

on boot



9/111

Stopping Postmaster

LOG: received smart shutdown request

LOG: autovacuum launcher shutting downLOG: shutting down

LOG: database system is shut down

manually

pg_ctl

on shutdown



10/111

Connections

local unix domain socket

host TCP/IP, both SSL or non-SSL

hostssl onlySSL

hostnossl never SSL



11/111

Authentication

trust

passwords md5 crypt password

remote authentication

host ident using pg_ident.conf

kerberos gss sspi



12/111

Authentication (continued)

local socket ident

host ident using local identd

socket permissions

pam

ldap

reject



13/111

Access

hostname and network mask

database name

role name (user or group)

filename or list of databases, role

IPv6



14/111

Pg_hba.conf

# TYPE DATABASE USER CIDR-ADDRESS METHOD

# "local" is for Unix domain socket connections onlylocal all all trust

# IPv4 local connections:

host all all 127.0.0.1/32 trust

# IPv6 local connections:host all all ::1/128 trust



15/111

Permissions

host connection permissions

role permissions

create roles create databases table permissions

Database creation

template1 customization system tables disk space computations



16/111

Data Directory

$ ls -CF

PG_VERSION pg_ident.conf pg_xlog/base/ pg_multixact/ postgresql.conf

global/ pg_subtrans/ postmaster.opts

pg_clog/ pg_tblspc/ postmaster.pid

pg_hba.conf pg_twophase/



17/111

Database Directories

$ ls -CF global/

1136 2397 2843

1137 2671 2844

1213 2672 2845

1214 2676 2846

1232 2677 2847

1233 2694 pg_auth

1260 2695 pg_control

1261 2697 pg_database1262 2698 pgstat.stat

2396 2842

$ ls -CF base/

1/ 11510/ 11511/ 16384/

$ ls -CF base/16384

11429 2617 2696

11431 2618 2699

...Mastering PostgreSQL Administration 16


18/111

Transaction/WAL Directories

$ ls -CF pg_xlog/

000000010000000000000000 archive_status/$ ls -CF pg_clog/

0000



19/111

Configuration Directories

$ ls -CF share/

conversion_create.sql postgres.description sql_features.txtinformation_schema.sql postgres.shdescription system_views.sql

pg_hba.conf.sample postgresql.conf.sample timezone/

pg_ident.conf.sample psqlrc.sample timezonesets/

pg_service.conf.sample recovery.conf.sample tsearch_data/

postgres.bki snowball_create.sql unknown.pltcl



20/111

Configuration Postgresql.conf



21/111

PostgreSQL.Conf

# -----------------------------

# PostgreSQL configuration file# -----------------------------

#

# This file consists of lines of the form:

#

# name = value#

# (The "=" is optional.) Whitespace may be used. Comments are introduced with

# "#" anywhere on a line. The complete list of parameter names and allowed

# values can be found in the PostgreSQL documentation.

## The commented-out settings shown in this file represent the default values.

# Re-commenting a setting is NOT sufficient to revert it to the default value;

# you need to reload the server.



22/111

PostgreSQL.Conf (Continued)

# This file is read on server startup and when the server receives a SIGHUP

# signal. If you edit the file on a running system, you have to SIGHUP the

# server for the changes to take effect, or use "pg_ctl reload". Some

# parameters, which are marked below, require a server shutdown and restart to

# take effect.

#

# Any parameter can also be given as a command-line option to the server, e.g.,# "postgres -c log_connections=on". Some paramters can be changed at run time

# with the "SET" SQL command.

#

# Memory units: kB = kilobytes MB = megabytes GB = gigabytes

# Time units: ms = milliseconds s = seconds min = minutes h = hours d = days



23/111

Configuration File Location

#data_directory = ConfigDir # use data in another directory

# (change requires restart)#hba_file = ConfigDir/pg_hba.conf # host-based authentication file

# (change requires restart)

#ident_file = ConfigDir/pg_ident.conf # ident configuration file


# If external_pid_file is not explicitly set, no extra PID file is written.#external_pid_file = (none) # write an extra PID file




24/111

Connections and Authentication

#listen_addresses = localhost # what IP address(es) to listen on;

# comma-separated list of addresses;# defaults to localhost, * = all


#port = 5432 # (change requires restart)

max_connections = 100 # (change requires restart)

# Note: Increasing max_connections costs ~400 bytes of shared memory per# connection slot, plus lock space (see max_locks_per_transaction). You might

# also need to raise shared_buffers to support more connections.

#superuser_reserved_connections = 3 # (change requires restart)

#unix_socket_directory = # (change requires restart)

#unix_socket_group = # (change requires restart)#unix_socket_permissions = 0777 # begin with 0 to use octal notation


#bonjour_name = # defaults to the computer name




25/111

Security and Authentication

#authentication_timeout = 1min # 1s-600s

#ssl = off # (change requires restart)#ssl_ciphers = ALL:!ADH:!LOW:!EXP:!MD5:@STRENGTH # allowed SSL ciphers


#password_encryption = on

#db_user_namespace = off

# Kerberos and GSSAPI#krb_server_keyfile = # (change requires restart)

#krb_-

srvname = postgres # (change requires restart, Kerberos only)

#krb_server_hostname = # empty string matches any keytab entry

# (change requires restart, Ker-beros only)

#krb_caseins_users = off # (change requires restart)

#krb_realm = # (change requires restart)



26/111

TCP/IP Control

# see "man 7 tcp" for details

#tcp_keepalives_idle = 0 # TCP_KEEPIDLE, in seconds;# 0 selects the system default

#tcp_keepalives_interval = 0 # TCP_KEEPINTVL, in seconds;

# 0 selects the system default

#tcp_keepalives_count = 0 # TCP_KEEPCNT;

# 0 selects the system default



27/111

Memory Usage

shared_buffers = 32MB # min 128kB or max_connections*16kB

# (change requires restart)#temp_buffers = 8MB # min 800kB

#max_prepared_transactions = 5 # can be 0 or more


# Note: Increasing max_prepared_transactions costs ~600 bytes of shared memory

# per transaction slot, plus lock space (see max_locks_per_transaction).#work_mem = 1MB # min 64kB

#maintenance_work_mem = 16MB # min 1MB

#max_stack_depth = 2MB # min 100kB

Kernel changes often required.



28/111

Sizing Shared Memory

MPage Out

Page In

R

A

Kernel Disk Buffer Cache

Free

Kernel

Postgres Backend

Postgres Backend

Postgres Backend

PostgreSQL Shared Buffer Cache

Swap



29/111

Free Space Map and Kernel Resources

# - Free Space Map -

max_fsm_pages = 204800 # min max_fsm_relations*16, 6 bytes each# (change requires restart)

#max_fsm_relations = 1000 # min 100, ~70 bytes each


# - Kernel Resource Usage -#max_files_per_process = 1000 # min 25


#shared_preload_libraries = # (change requires restart)



30/111

Vacuum and Background Writer

# - Cost-Based Vacuum Delay -

#vacuum_cost_delay = 0 # 0-1000 milliseconds#vacuum_cost_page_hit = 1 # 0-10000 credits

#vacuum_cost_page_miss = 10 # 0-10000 credits

#vacuum_cost_page_dirty = 20 # 0-10000 credits

#vacuum_cost_limit = 200 # 1-10000 credits

# - Background Writer -

#bgwriter_delay = 200ms # 10-10000ms between rounds

#bgwriter_lru_maxpages = 100 # 0-1000 max buffers written/round

#bgwriter_lru_multiplier = 2.0 # 0-

10.0 multipler on buffers scanned/round



31/111

Write-Ahead Log (WAL)

#fsync = on # turns forced synchronization on or off

#synchronous_commit = on # immediate fsync at commit

#wal_sync_method = fsync # the default is the first option

# supported by the operating system:# open_datasync

# fdatasync

# fsync

# fsync_writethrough

# open_sync#full_page_writes = on # recover from partial page writes

#wal_buffers = 64kB # min 32kB


#wal_writer_delay = 200ms # 1-10000 milliseconds

#commit_delay = 0 # range 0-100000, in microseconds#commit_siblings = 5 # range 1-1000



32/111

Write-Ahead Logging (Continued)

fsync

fsync

BackendPostgres

BackendPostgres

BackendPostgres

PostgreSQL Shared Buffer Cache WriteAhead Log

Kernel Disk Buffer Cache

Disk Blocks



33/111

Checkpoints and Archiving

# - Checkpoints -

#checkpoint_segments = 3 # in logfile segments, min 1, 16MB each

#checkpoint_timeout = 5min # range 30s-1h

#checkpoint_completion_target = 0.5 # checkpoint target duration, 0.0 - 1.0#checkpoint_warning = 30s # 0 is off

# - Archiving -

#archive_mode = off # allows archiving to be done


#archive_command = # command to use to archive a logfile segment

#archive_timeout = 0 # force a logfile segment switch after this

# time; 0 is off



34/111

Query Tuning

# - Planner Method Configuration -

#enable_bitmapscan = on

#enable_hashagg = on

#enable_hashjoin = on#enable_indexscan = on

#enable_mergejoin = on

#enable_nestloop = on

#enable_seqscan = on

#enable_sort = on#enable_tidscan = on

# - Planner Cost Constants -

#seq_page_cost = 1.0 # measured on an arbitrary scale

#random_page_cost = 4.0 # same scale as above#cpu_tuple_cost = 0.01 # same scale as above

#cpu_index_tuple_cost = 0.005 # same scale as above

#cpu_operator_cost = 0.0025 # same scale as above

#effective_cache_size = 128MB



35/111

Query Tuning (Continued)

#geqo = on

#geqo_threshold = 12#geqo_effort = 5 # range 1-10

#geqo_pool_size = 0 # selects default based on effort

#geqo_generations = 0 # selects default based on effort

#geqo_selection_bias = 2.0 # range 1.5-2.0

# - Other Planner Options -

#default_statistics_target = 10 # range 1-1000

#constraint_exclusion = off

#from_collapse_limit = 8

#join_collapse_limit = 8 # 1 disables collapsing of explicit# JOIN clauses



36/111

Where To Log

#log_destination = stderr # Valid values are combinations of

# stderr, csvlog, syslog and eventlog,# depending on platform. csvlog

# requires logging_collector to be on.

# This is used when logging to stderr:

#logging_collector = off # Enable capturing of stderr and csvlog

# into log files. Required to be on for# csvlogs.


# These are only used if logging_collector is on:

#log_directory = pg_log # directory where log files are written,

# can be absolute or relative to PGDATA#log_filename = postgresql-%Y-%m-%d_%H%M%S.log # log file name pattern,

# can include strftime() escapes



37/111

Where To Log (Continued)

#log_truncate_on_rotation = off # If on, an existing log file of the

# same name as the new log file will be

# truncated rather than appended to.# But such truncation only occurs on

# time-driven rotation, not on restarts

# or size-driven rotation. Default is

# off, meaning append to existing files

# in all cases.#log_rotation_age = 1d # Automatic rotation of logfiles will

# happen after that time. 0 to disable.

#log_rotation_size = 10MB # Automatic rotation of logfiles will

# happen after that much log output.

# 0 to disable.

# These are relevant when logging to syslog:

#syslog_facility = LOCAL0

#syslog_ident = postgres



38/111

When to Log

#client_min_messages = notice # values in order of decreasing detail:

# debug5

# debug4

# debug3

# debug2

# debug1

# log

# notice

# warning

# error

#log_min_messages = notice # values in order of decreasing detail:

# debug5

# debug4

# debug3

# debug2

# debug1

# info

# notice

# warning

# error

# log

# fatal

# panic



39/111

When to Log (Continued)

#log_error_verbosity = default # terse, default, or verbose messages

#log_min_error_statement = error # values in order of decreasing detail:

# debug5

# debug4

# debug3# debug2

# debug1

# info

# notice

# warning

# error# log

# fatal

# panic (effectively off)

#log_min_duration_statement = -1 # -1 is disabled, 0 logs all statements

# and their durations, > 0 logs only

# statements running at least this time.#silent_mode = off # DO NOT USE without syslog or

# logging_collector




40/111

What to Log

#debug_print_parse = off

#debug_print_rewritten = off#debug_print_plan = off

#debug_pretty_print = off

#log_checkpoints = off

#log_connections = off

#log_disconnections = off#log_duration = off

#log_hostname = off



41/111

What To Log: Log_line_prefix

#log_line_prefix = # special values:

# %u = user name

# %d = database name# %r = remote host and port

# %h = remote host

# %p = process ID

# %t = timestamp without milliseconds

# %m = timestamp with milliseconds# %i = command tag

# %c = session ID

# %l = session line number

# %s = session start timestamp

# %v = virtual transaction ID# %x = transaction ID (0 if none)

# %q = stop here in non-session

# processes

# %% = %

# e.g. Mastering PostgreSQL Administration 40


42/111

What to Log (Continued)

#log_lock_waits = off # log lock waits >= deadlock_timeout

#log_statement = none # none, ddl, mod, all

#log_temp_files = -1 # log temporary files equal or larger

# than specified size;

# -1 disables, 0 logs all temp files

#log_timezone = unknown # actually, defaults to TZ environment

# setting



43/111

Runtime Statistics

# - Query/Index Statistics Collector -

#track_activities = on

#track_counts = on

#update_process_title = on

# - Statistics Monitoring -

#log_parser_stats = off#log_planner_stats = off

#log_executor_stats = off

#log_statement_stats = off



44/111

Autovacuum

#autovacuum = on # Enable autovacuum subprocess? on

# requires track_counts to also be on.

#log_autovacuum_min_duration = -1 # -1 disables, 0 logs all actions and

# their durations, > 0 logs only

# actions running at least that time.#autovacuum_max_workers = 3 # max number of autovacuum subprocesses

#autovacuum_naptime = 1min # time between autovacuum runs

#autovacuum_vacuum_threshold = 50 # min number of row updates before

# vacuum

#autovacuum_analyze_threshold = 50 # min number of row updates before

# analyze#autovacuum_vacuum_scale_factor = 0.2 # fraction of table size before vacuum

#autovacuum_analyze_scale_factor = 0.1 # fraction of table size before analyze

#autovacuum_freeze_max_age = 200000000 # maximum XID age before forced vacuum


#autovacuum_vacuum_cost_delay = 20 # default vacuum cost delay for

# autovacuum, -1 means use# vacuum_cost_delay

#autovacuum_vacuum_cost_limit = -1 # default vacuum cost limit for

# autovacuum, -1 means use

# vacuum_cost_limit



45/111

Statement Behavior

#search_path = "$user",public # schema names

#default_tablespace = # a tablespace name, uses the default

#temp_tablespaces = # a list of tablespace names, uses

# only default tablespace

#check_function_bodies = on

#default_transaction_isolation = read committed

#default_transaction_read_only = off#session_replication_role = origin

#statement_timeout = 0 # 0 is disabled

#vacuum_freeze_min_age = 100000000

#xmlbinary = base64

#xmloption = content



46/111

Locale and Formatting

datestyle = iso, mdy

#timezone = unknown # actually, defaults to TZ environment

# setting

#timezone_abbreviations = Default # Select the set of available time zone

# abbreviations. Currently, there are# Default

# Australia

# India

# You can create your own file in

# share/timezonesets/.#extra_float_digits = 0 # min -15, max 2

#client_encoding = sql_ascii # actually, defaults to database

# encoding

# These settings are initialized by initdb, but they can be changed.

lc_messages = C # locale for system error message# strings

lc_monetary = C # locale for monetary formatting

lc_numeric = C # locale for number formatting

lc_time = C # locale for time formatting



47/111

Full Text Search

# default configuration for text search

default_text_search_config = pg_catalog.english



48/111

Other Defaults

#explain_pretty_print = on

#dynamic_library_path = $libdir

#local_preload_libraries =



49/111

Lock Management

#deadlock_timeout = 1s

#max_locks_per_transaction = 64 # min 10


# Note: Each lock table slot uses ~270 bytes of shared memory, and there are

# max_locks_per_transaction * (max_connections + max_prepared_transactions)

# lock table slots.



50/111

Version/Platform Compatibility

# - Previous PostgreSQL Versions -

#add_missing_from = off

#array_nulls = on

#backslash_quote = safe_encoding # on, off, or safe_encoding

#default_with_oids = off

#escape_string_warning = on

#regex_flavor = advanced # advanced, extended, or basic

#sql_inheritance = on

#standard_conforming_strings = off

#synchronize_seqscans = on

# - Other Platforms and Clients -#transform_null_equals = off


C i i


51/111

Customization

#custom_variable_classes = # list of custom variable class names


I f


52/111

Interfaces

Installing

Compiled Languages (C, ecpg) Scripting Language (Perl, Python) SPI

Connection Pooling


I l d Fil


53/111

Include Files

$ ls -CF include/

ecpg_config.h libpq/ pgtypes_interval.h

ecpg_informix.h libpq-fe.h pgtypes_numeric.h

ecpgerrno.h pg_config.h pgtypes_timestamp.h

ecpglib.h pg_config_manual.h postgres_ext.h

ecpgtype.h pg_config_os.h server/

informix/ pgtypes_date.h sql3types.hinternal/ pgtypes_error.h sqlca.h


Lib Fil


54/111

Library Files

$ ls -CF lib/

ascii_and_mic.so* libecpg_compat.so.3@ utf8_and_big5.so*

cyrillic_and_mic.so* libecpg_compat.so.3.0* utf8_and_cyrillic.so*dict_snowball.so* libpgport.a utf8_and_euc_cn.so*

euc_cn_and_mic.so* libpgtypes.a utf8_and_euc_jis_2004.s

euc_jis_2004_and_shift_ libpgtypes.so@ utf8_and_euc_jp.so*

euc_jp_and_sjis.so* libpgtypes.so.3@ utf8_and_euc_kr.so*

euc_kr_and_mic.so* libpgtypes.so.3.0* utf8_and_euc_tw.so*euc_tw_and_big5.so* libpq.a utf8_and_gb18030.so*

latin2_and_win1250.so* libpq.so@ utf8_and_gbk.so*

latin_and_mic.so* libpq.so.5@ utf8_and_iso8859.so*

libecpg.a libpq.so.5.1* utf8_and_iso8859_1.so*

libecpg.so@ pgxs/ utf8_and_johab.so*libecpg.so.6@ plperl.so* utf8_and_shift_jis_2004

libecpg.so.6.0* plpgsql.so* utf8_and_sjis.so*

libecpg_compat.a pltcl.so* utf8_and_uhc.so*

libecpg_compat.so@ utf8_and_ascii.so* utf8_and_win.so*


Maintenance


55/111

Maintenance


Backup


56/111

Backup

File system-level

tar, cpio while shutdown file system snapshot rsync, shutdown, rsync, restart

pg_dump/pg_dumpall

restore/pg_restore with custom format


Continuous Archiving /


57/111

Continuous Archiving /

Point-In-Time Recovery (PITR)

WALWAL

WAL

Continuous

Backup

0200

1100

0900

1300

File System-

Level Backup


PITR Backup Procedures


58/111

PITR Backup Procedures

1. archive_command = cp %p /mnt/server/pgsql/%f

2. SELECT pg_start_backup(label);

3. Perform file system-level backup (can be inconsistent)

4. SELECT pg_stop_backup();


PITR Recovery


59/111

PITR Recovery

WAL1

740

1

755

Continuous

Backup

File System-

Level Backup

WAL

WAL

1

700

1

730


PITR Recovery Procecdures


60/111

PITR Recovery Procecdures

1. Stop postmaster

2. Restore file system-level backup

3. Make adjustments as outlined in the documentation

4. Create recovery.conf

5. Add restore_command = cp /mnt/server/pgsql/%f %p

6. Start the postmaster


Master-Slave Replication - Slony


61/111

Master Slave Replication Slony

! ! ! !

! ! ! !

! ! ! !

" " " "

" " " "

# # # #

# # # #

$ $ $ $ $

$ $ $ $ $

$ $ $ $ $

$ $ $ $ $

$ $ $ $ $

$ $ $ $ $

% % % % %

% % % % %

% % % % %

% % % % %

% % % % %

& & & &

& & & &

& & & &

& & & &

' ' ' '

' ' ' '

' ' ' '

( ( ( (

( ( ( (

) ) ) )

) ) ) )

0 0 0 0 0

0 0 0 0 0

0 0 0 0 0

0 0 0 0 0

0 0 0 0 0

0 0 0 0 0

1 1 1 1 1

1 1 1 1 1

1 1 1 1 1

1 1 1 1 1

1 1 1 1 1

2 2 2 2

2 2 2 2

2 2 2 2

3 3 3 3

3 3 3 3

3 3 3 3

4 4 4 4

4 4 4 4

5 5 5 5

5 5 5 5

6

6

6

6

6

6

6

6

6

6

6

6

6

6

8

8

8

8

8

8

8

8

8

8

8

8

8

8

9

9

9

9

9

9

9

9

9

9

9

9

9

9

@

@

@

@

@

@

@

@

@

@

@

@

@

@

A

A

A

A

A

A

A

A

A

A

A

A

A

A

B

B

B

B

B

B

B

B

B

B

B

B

B

B

C

C

C

C

C

C

C

C

C

C

C

C

C

C

D

D

D

D

D

D

D

D

D

D

D

D

D

D

E

E

E

E

E

E

E

E

E

E

E

E

E

E

F

F

F

F

F

F

F

F

F

F

F

F

F

F

G

G

G

G

G

G

G

G

G

G

G

G

G

G

H

H

H

H

H

H

H

H

H

H

H

H

H

H

I

I

I

I

I

I

I

I

I

I

I

I

I

I

P

P

P

P

P

P

P

P

P

P

P

P

P

P

Q

Q

Q

Q

Q

Q

Q

Q

Q

Q

Q

Q

Q

Q

R

R

R

R

R

R

R

R

R

R

R

R

R

R

S

S

S

S

S

S

S

S

S

S

S

S

S

S

T

T

T

T

T

T

T

T

T

T

T

T

T

T


Other Solutions


62/111

Other Solutions

Mutli-master replication: pgcluster, bacardo

Pooling: pgpool


Data Maintenance


63/111

VACUUM (nonblocking), free space map

VACUUM FULL

ANALYZE


Vacuum


64/111

Shared Memory

Hashed

Free Space Map

Block #

DB oid

Block #

Block #Table

Relfilenode

Block #

Table

Block #

Block #Block # Block #

Table


Vacuum Full


65/111

V

A

C

T

E

IV

A

C

T

E

IV

A

C

T

E

IV

A

C

T

E

IV

A

C

T

E

I

V

A

C

T

E

I

V

E

X

P

E

I

R

A

C

T

E

I

V

A

C

T

E

I

V

A

C

T

E

I

V

A

C

T

E

I

V

A

C

T

E

I

V

A

C

T

E

I

V

AC

T

E

I

V

AC

T

E

I

V

AC

T

E

I

V

AC

T

E

I

V

AC

T

E

I

V

AC

T

E

I

V

AC

T

E

I

V

AC

T

E

I

V

AC

T

E

I

V

AC

T

E

I

V

A

C

T

E

I

V

E

X

P

E

I

R

A

C

T

E

I

V

Truncate File

SlotsRows Into Expired

Move Trailing

Rows Identified

With Expired

Original Heap

A

C

T

E

IV

A

C

T

E

IV

A

C

T

E

IV

A

C

T

E

IV

A

C

T

E

IV

A

C

T

E

I


Checkpoints


66/111

p

Write all dirty shared buffers

Sync all dirty kernel buffers

Recycle WAL files

Check for server messages indicating too-frequent checkpoints

If so, increase checkpoint_segments


Automating Tasks


67/111

g

0 3 * * * root psql -c VACUUM FULL; test

0 3 * * * root vacuumdb -a -f

Autovacuum usually makes vacuum administration unnecessary.


Monitoring Active Sessions


68/111


ps


69/111

$ ps -Upostgres

PID TT STAT TIME COMMAND

2125 ?? Ss 0:00.26 ./bin/postmaster -i2142 ?? S 0:00.03 stats buffer process (postmaster)

2143 ?? S 0:00.06 stats collector process (postmaster)

3341 ?? I 0:00.07 postgres test [local] idle (postmaster)

3340 p6 I+ 0:00.03 psql test


top


70/111

$ top

load averages: 0.56, 0.39, 0.36 18:25:58

138 processes: 5 running, 130 sleeping, 3 zombieCPU states: 50.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 50.0% idle

Memory: Real: 96M/133M Virt: 535M/1267M Free: 76M

PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND

23785 postgres 57 0 11M 5336K run/0 0:07 30.75% 30.66% postmaster23784 postgres 2 0 10M 11M sleep 0:00 2.25% 2.25% psql


pgmonitor


71/111


Query Monitoring


73/111


Access Statistics


77/111

_ |

last_autovacuum |last_analyze |

last_autoanalyze |


Table Block Activity


79/111

Heavily used tables

Unnecessary indexes

Additional indexes

Index usage

TOAST usage


CPU


80/111

$ vmstat 5

procs memory page disks faults cpu

r b w avm fre flt re pi po fr sr s0 s0 in sy cs us sy id1 0 0 501820 48520 1234 86 2 0 0 3 5 0 263 2881 599 10 4 86

3 0 0 512796 46812 1422 201 12 0 0 0 3 0 259 6483 827 4 7 88

3 0 0 542260 44356 788 137 6 0 0 0 8 0 286 5698 741 2 5 94

4 0 0 539708 41868 576 65 13 0 0 0 4 0 273 5721 819 16 4 80

4 0 0 547200 32964 454 0 0 0 0 0 5 0 253 5736 948 50 4 464 0 0 556140 23884 461 0 0 0 0 0 2 0 249 5917 959 52 3 44

1 0 0 535136 46280 1056 141 25 0 0 0 2 0 261 6417 890 24 6 70


I/O


81/111

$ iostat 5

tty sd0 sd1 sd2 % cpu

tin tout sps tps msps sps tps msps sps tps msps usr nic sys int idl7 119 244 11 6.1 0 0 27.3 0 0 18.1 9 1 4 0 86

0 86 20 1 1.4 0 0 0.0 0 0 0.0 2 0 2 0 96

0 82 61 4 3.6 0 0 0.0 0 0 0.0 2 0 2 0 97

0 65 6 0 0.0 0 0 0.0 0 0 0.0 1 0 2 0 97

12 90 31 2 5.4 0 0 0.0 0 0 0.0 4 0 3 0 9324 173 6 0 4.9 0 0 0.0 0 0 0.0 48 0 3 0 49

0 91 3594 63 4.6 0 0 0.0 0 0 0.0 11 0 4 0 85


Disk Usage


82/111

play=# SELECT relfilenode, relpages * 8 AS kilobytes

play-# FROM pg_class

play-# WHERE relname = customer;relfilenode | kilobytes

-------------+----------

16806 | 480

(1 row)

Vacuum required. dbsize available.


TOAST Usage


83/111

play=# SELECT relname, relpages * 8 AS kilobytes


play-# WHERE relname = pg_toast_16806 ORplay-# relname = pg_toast_16806_index

play-# ORDER BY relname;

relname | kilobytes

----------------------+----------

pg_toast_16806 | 0pg_toast_16806_index | 1


Index Usage


84/111

play=# SELECT c2.relname, c2.relpages * 8 AS kilobytes

play-# FROM pg_class c, pg_class c2, pg_index i

play-# WHERE c.relname = customer ANDplay-# c.oid = i.indrelid AND

play-# c2.oid = i.indexrelid

play-# ORDER BY c2.relname;

relname | kilobytes

----------------------+----------customer_id_indexdex | 26


Largest Tables


85/111

play=# SELECT relname, relpages * 8


play-# ORDER BY relpages DESC;relname | kilobytes

----------------------+----------

bigtable | 3290

customer | 3144


Database File Mapping - oid2name


86/111

$ oid2name

All databases:

---------------------------------18720 = test1

1 = template1

18719 = template0

18721 = test

18735 = postgres18736 = cssi


Table File Mapping


87/111

$ cd /usr/local/pgsql/data/base

$ oid2name

All databases:---------------------------------

16817 = test2

16578 = x

16756 = test

1 = template116569 = template0

16818 = test3

16811 = floattest

$ cd 16756$ ls 1873*

18730 18731 18732 18735 18736 18737 18738 18739



88/111

$ oid2name -d test -o 18737

Tablename of oid 18737 from database "test":

---------------------------------

18737 = ips

$ oid2name -d test -t ips

Oid of table ips from database "test":

---------------------------------

18737 = ips

$ # show disk space for every db object

$ du * | while read SIZE RELFILENODE

> do

> echo "$SIZE oid2name -q -d test -o $RELFILENODE"

> done24 18737 = ips

36 18722 = cities

...


$ # b b t t b l t fi t


89/111

$ # same as above, but sort by largest first$ du * | while read SIZE OID

> do

> echo "$SIZE oid2name -q -d test -o $OID"

> done |

> sort -rn2048 19324 = bigtable

1950 23903 = customers

...

$ # show disk usage per database

$ cd /usr/local/pgsql/data/base

$ du -s * |

> while read SIZE OID

> do

> echo "$SIZE oid2name -q | grep ^$OID "

> done |

> sort -rn

2256 18721 = test

2135 18735 = postgresMastering PostgreSQL Administration 88

Disk Balancing


90/111

Move pg_xlog to another drive using symlinks

Tablespaces


Per-Database Tablespaces


91/111

DB4

Disk 3

DB2 DB3DB1

Disk 1 Disk 2


Per-Object Tablespaces


92/111

constrainttab1 tab2 index

Disk 1 Disk 2 Disk 3


Analyzing Locking

$ ps -Upostgres


93/111

$ ps -Upostgres

PID TT STAT TIME COMMAND

9874 ?? I 0:00.07 postgres test [local] idle in transaction (postmaster)

9835 ?? S 0:00.05 postgres test [local] UPDATE waiting (postmaster)

10295 ?? S 0:00.05 postgres test [local] DELETE waiting (postmaster)

test=> SELECT * FROM pg_locks;

relation | database | transaction | pid | mode | granted

----------+----------+-------------+------+------------------+---------

17143 | 17142 | | 9173 | AccessShareLock | t17143 | 17142 | | 9173 | RowExclusiveLock | t

| | 472 | 9380 | ExclusiveLock | t

| | 468 | 9338 | ShareLock | f


16759 | 17142 | | 9380 | AccessShareLock | t17143 | 17142 | | 9338 | AccessShareLock | t

17143 | 17142 | | 9338 | RowExclusiveLock | t


(9 rows)


Miscellaneous Tasks


94/111

Log file rotation, syslog

Upgrading

Migration


Administration Tools


95/111

pgadmin

phppgadmin


Recovery


96/111


Client Application Crash


97/111

Nothing Required. Transactions in progress are rolled back.


Graceful Server Crash


98/111



Abrupt Server Crash


99/111



Operating System Crash


100/111

Nothing Required. Transactions in progress are rolled back. Partial pagewrites are repaired.


Disk Failure


101/111

Restore from previous backup or use PITR.


Accidental DELETE


102/111

Recover table from previous backup, perhaps using pg_restore. It ispossible to modify the backend code to make deleted tuples visible,dump out the deleted table and restore the original code. All tuples inthe table since the previous vacuum will be visible. It is possible torestrict that so only tuples deleted by a specific transaction are visible.


Write-Ahead Log (WAL) Corruption


103/111

See pg_resetxlog. Review recent transactions and identify any damage,including partially committed transactions.


File Deletion


104/111

It may be necessary to create an empty file with the deleted file name sothe object can be deleted, and then the object restored from backup.


Accidental DROP TABLE


105/111

Restore from previous backup.


Accidental DROP INDEX


106/111

Recreate index.


Accidental DROP DATABASE


107/111

Restore from previous backup.


Non-Starting Installation


108/111

Restart problems are usually caused by write-ahead log problems. Seepg_resetxlog. Review recent transactions and identify any damage,including partially committed transactions.


Index Corruption


109/111

Use REINDEX.


Table Corruption


110/111

Try reindexing the table. Try identifying the corrupt OID of the row andtransfer the valid rows into another table using SELECTINTOWHEREoid != ###. Use http://sources.redhat.com/rhdb/tools.html to analyzethe internal structure of the table.


Conclusion
http://sources.redhat.com/rhdb/tools.htmlhttp://sources.redhat.com/rhdb/tools.html


111/111


masterpgadministration-091023104603-phpapp01

Documents

Transcript of masterpgadministration-091023104603-phpapp01