Why we love pgpool-II and why we hate it!
-
Upload
pgday-asia -
Category
Software
-
view
261 -
download
4
Transcript of Why we love pgpool-II and why we hate it!
Why we love pgpooland why hate it!
Sameer Kumar (@sameerkasi200x)
Sr. Solution Architect, Ashnik (@AshnikBiz)
About Me!
• A Random guy who started Career as an Oracle and DB2 DBA (and yeah a bit of SQL Server too)
• Then moved to ‘Ashnik’ and started working with Postgres
• We work on Open Source Consulting and Solution
• And now I love Open Source!
• I also work on• Backend solution for Mobile App• Web Infrastructure• Docker and Containers
• Twitter - @sameerkasi200x
• Apart from technology I love cycling and photography
What am I going to talk?
• What is pgpool?
• What got us hooked to pgpool?
• What architectures we deployed with pgpool?
• Issues we run into• Some diagnostics and resolutions
• Some tips?
• IMHO, when not to use pgpool?
What got us hooked to pgpool?
What is pgpool?
• pgpool-II is a middleware
• Works between PostgreSQL servers and a PostgreSQL database client
• Licensed under BSD license
pgpool is Swiss Army knife for PostgreSQL!
• It offers various features• Connection Pooling
• Replication
• Load Balancing
• Limiting Exceeding Connections
• Parallel Query (deprecated)
Connection Pooling
• pgpool-II saves connections to the PostgreSQL servers
• reuse them when a new connection with the same properties
• It reduces connection overhead, and improves system's overall throughput
Replication
• pgpool-II can manage multiple PostgreSQL servers
• Using the replication function enables creating a realtimebackup on 2 or more physical disks
Load Balancing
Load Balancing
• Read query is automatically load balanced
• pgpool can detect failover and start sending Read/write to surviving node
• It can also be used to run a failover command to promote standby upon failure of master
Watchdog Mode
Limiting Exceeding Connections a.k.a Connection Queuing
• In PostgreSQL connections after max_connections are rejected
• pgpool-II also has a limit on the maximum number of connection
• Extra connections will be queued
What got us hooked to pgpool?
• It understands the PostgreSQL wire protocol and has Postgres query parser?
• In plain English it means• You don’t have to do major changes in your application to use features
like load balancing and connections pooling.
• Most of the times you can use it with pgpool literally unchanged
• Makes it perfect• While scaling after development
• Using packaged application
We really started loving it!
• More stand-by servers can be added and pgpool can be configured for load balancing across more nodes in runtime
• Works in tandem with Virtualization and Provisioning on the fly
• Has in-built capability for HA with Watchdog
Architectures we deployed with pgpool?
Streaming Replication – Master Slave
Streaming Replication – with Load Balancing
Load Balancing – with third party failover solution
Load Balancing with pgpool Watchdog– a.k.a. NoSPoF
Pgpool tied with Application Servers
Issues we faced!
Does not work very well with application connection pooling!
An I/O error occured while sending to the backend.
A PooledConnection that has already signalled a Connection error is still in use
com.edb.util.PSQLException: This connection has been closed.
• Occurs when a connection was idle in your application pool but was closed by pgpool
• Also when using application connection pooling, less changes of pgpool’s connection pool being useful
It is not guaranteed that pooled connections will be used
• In pgpool connections are pooled by child processes
• Each child process can pool multiple connections if they have different user-db combination
• When pgpool receives a request it assigns it to any child – “any child” randomly
• It is not guaranteed that a child pooling a relevant connection will receive request
We hit several issues and known bugs over 6 months
connect_inet_domain_socket: select() interrupted
pid 22001: die: close listen socket
pcp_child: pcp_read() failed. reason: Success
pcp_child: authentication failed
an error message: ERROR: unable to read data from frontend DETAIL: EOF encountered with frontend
Logging is not really best for someone who has not looked at the code
pid 22001: die: close listen socket
pcp_child: pcp_read() failed. reason: Success
health_check: health check timer has been already expired before attempting to connect to 1 th backend
Split Brain in Master Slave configuration
• False promotion of the Standby node during a network split
• No way to define quorum or voting before deciding to promote
• No auto reconfiguration of replication if you have multiple standby
You can not use pg_terminate_backend()
• pg_terminate_backend sends a signal back to the client, which is very similar to response sent when server is being shutdown
• pgpool uses the response from Server on connections determine it’s health
• pg_terminate_backend
No way to poll lost nodes automatically
• If a node is removed from pgpool, there is no check done by pgpool to check if it is ready again
• A periodic check to check if the same node is available again, would be nice
• This is specially good when you use a separate tool for failovers • Imagine when failover did not happen but pgpool thought master has
gone down
Connection restart upon node failure
2016-08-13 03:17:20 LOG: pid 8662: health_check: health check timer has been already expired before attempting to connect to 1 th backend
2016-08-13 03:17:20 LOG: pid 8662: set 1 th backend down status
2016-08-13 03:17:20 LOG: pid 8662: wd_start_interlock: start interlocking
2016-08-13 03:17:20 LOG: pid 8662: starting degeneration. shutdown host 172.163.169.150(5432)
2016-08-13 03:17:20 LOG: pid 8662: Restart all children
.
.
2016-08-13 03:17:34 LOG: pid 16338: pcp child process received restart request
2016-08-13 03:17:34 LOG: pid 8662: PCP child 16338 exits with status 256 in failover()
2016-08-13 03:17:34 LOG: pid 8662: fork a new PCP child pid 5133 in failover()
2016-08-13 03:18:11 LOG: pid 9204: worker process received restart request
Time skew can lead to issues in Watchdog mode
• pgpool exchanges UDP messages to keep a tab on other health of other pgpoolnodes
• These UDP messages contain timestamp
• If server time deviates or gets synchronized, there is a chance that pgpool will discard messages thinking it is from past
(!WD_TIME_ISSET(node->hb_send_time) ||WD_TIME_BEFORE(node->hb_send_time, pkt.send_time)){ereport(DEBUG1,(errmsg("received heartbeat signal from \"%s:%d\"",from, from_pgpool_port)));
node->hb_send_time = pkt.send_time;node->hb_last_recv_time = tv;}else{ereport(NOTICE,(errmsg("received heartbeat signal is older than the latest, ignored")));
Some tips!
• When running in Virtual Environment reserve the resource for Virtual Machine – i.e. no ballooning
• Make sure your pgpool timeout for node failover are higher than the failover
• Make sure you understand the authentication flow define pg_hba and pool_hba rule accordingly• Application --- AUTHENTICATES AGAINST pgpool --- AUTHENTICATES AGAINST PostgreSQL
• There are few hidden (not documented) checks that are done by pgpool• good to allow connections from pgpool to PostgreSQL for system
database e.g. postgres, template0 etc
Some tips while running pgpool
• Ensure that your servers hosting pgpool are in synchronized for their time
• pgpool timeout for node failover > than max time lag allowed / time sync frequency in your environment
• Use a different user for pgpool health check – with read only permissions/restrictions
• Learn to use pcp_* commands – manually and with your failover tool
Some more tips
• You have huge number of connections – any node failure can lead to restart of connections leading to a disconnection
• When you want to use it just for connection pooling – use pgbouncer instead
• When you want lost nodes to join back automatically
• When you are operating in a network where latency and QoS is not guaranteed
• You want queries to continue during a failure
• When you want to use it with different DB clusters
IMHO pgpool is not best when
Twitter - @sameerkasi200x | @ashnikbiz
Email - [email protected] | [email protected]
LinkedIn - https://www.linkedin.com/in/samkumar150288
We are hiring!