7 Ways To Crash Postgres
-
Upload
postgresql-experts-inc -
Category
Technology
-
view
4.497 -
download
1
description
Transcript of 7 Ways To Crash Postgres
![Page 1: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/1.jpg)
waysto
crashPostgres
7
Josh BerkusPostgreSQL ProjectLinuxConf.AU 2013
![Page 2: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/2.jpg)
![Page 3: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/3.jpg)
![Page 4: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/4.jpg)
Don'tApplyUpdates1
![Page 5: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/5.jpg)
1. ignore update announcements
2.keep running version X.X.0 or X.X.1
3.wait for disaster to strike
![Page 6: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/6.jpg)
9. 2. 2
![Page 7: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/7.jpg)
9. 2. 3
![Page 8: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/8.jpg)
![Page 9: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/9.jpg)
9.2.2:Prevent database-wide crash and restart when a new child process is unable to create a pipe for its latch.
9.1.6:Fix misbehavior when default_transaction_isolation is set to serializable. Symptoms include crashes at process start on Windows, and crashes in hot standby operation.
![Page 10: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/10.jpg)
Upgrade!
EOL after 5 years
![Page 11: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/11.jpg)
8.3 is now EOL
![Page 12: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/12.jpg)
Out ofDiskSpace2
![Page 13: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/13.jpg)
1. install DB on volume which is 80% full
2.don't monitor disk space usage
3. let database grow
![Page 14: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/14.jpg)
ERROR: could not write block 31840050 of temporary file: No space left on device
PANIC: could not write to file"pg_xlog/xlogtemp.49680": No space left on device
![Page 15: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/15.jpg)
Out ofDisk SpaceArchivingEdition
2a
![Page 16: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/16.jpg)
1.set up transaction log archiving to a standby
2.stop file use/deletion on the standby
3.don't monitor
![Page 17: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/17.jpg)
4. disk on the standby fills up with archive logs
5. disk on the master fills up with archive logs
6. failover impossible
![Page 18: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/18.jpg)
freeing up spacetakes more space
● VACUUM FULL● REINDEX● DELETE● pg_reorg
![Page 19: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/19.jpg)
DROP TABLE;DROP DATABASE;
![Page 20: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/20.jpg)
DeletingStuff3
![Page 21: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/21.jpg)
base pg_snapshotscold pg_stat_tmpglobal pg_subtranshot pg_tblspcpg_clog pg_twophasepg_hba.conf PG_VERSIONpg_ident.conf pg_xlogpg_log postgresql.confpg_multixact pg_notify postmaster.optspg_serial
![Page 22: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/22.jpg)
base pg_snapshotscold pg_stat_tmpglobal pg_subtranshot pg_tblspcpg_clog pg_twophasepg_hba.conf PG_VERSIONpg_ident.conf pg_xlogpg_log postgresql.confpg_multixact pg_notify postmaster.optspg_serial
![Page 23: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/23.jpg)
2012-09-25 08:27:03 UTC ERROR: could not access status of transaction 500185903
2012-09-25 08:27:03 UTC DETAIL: Could not open file "pg_clog/01DD": No such file or directory.
![Page 24: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/24.jpg)
Out ofMemory4
![Page 25: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/25.jpg)
1.set shared_buffers to 80% of RAM
2.set work_mem to 1GB
3.ramp up to 1000connections
![Page 26: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/26.jpg)
1.run Postgres on cloud server
2.add a JVM
3.add Apache/mod_php
4.add MySQL
5.add a second Postgres version
![Page 27: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/27.jpg)
0. run server with defaultLinux memory management settings
![Page 28: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/28.jpg)
kernel: postgres invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
LOG: background writer process (PID 22999) was terminated by signal 9: Killed
LOG: terminating any other active server processes
![Page 29: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/29.jpg)
BadHardware5
![Page 30: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/30.jpg)
1.buy brand-new hardware
2. install PostgreSQL
3.don't test
4.deploy in production
![Page 31: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/31.jpg)
1.deploy production hardware
2.don't monitor SMART
3.don't monitor syslog
4.run for a couple years
![Page 32: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/32.jpg)
ERROR: could not read block 1 in file "base/18511/13872": read only 0 of 8192 bytes
FATAL: failed to re-find parent key in index "257969064" for split pages8366/12375
![Page 33: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/33.jpg)
TooManyConnections6
![Page 34: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/34.jpg)
1.build application with ad-hoc DB connections
2.no connection pooling
3.program auto-reconnect on timeout
4.raise max_connections to 2000
![Page 35: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/35.jpg)
CRITICAL: server postgres2 is not accepting connections
![Page 36: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/36.jpg)
ZombieLocks7
![Page 37: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/37.jpg)
1.program application so it forgets to close transactions
2.run for a few hours or days
3.watch the locks pile up
![Page 38: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/38.jpg)
DBI ERROR timeout after 300 seconds: "UPDATE sessions SET last_seen = now()WHERE session_key = 'a16g32b12cd0';"
![Page 39: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/39.jpg)
1. no updates
2. out of disk space
3. deleting stuff
4. out of RAM
5. bad hardware
6. too many connections
7. zombie locks
![Page 40: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/40.jpg)
![Page 41: 7 Ways To Crash Postgres](https://reader033.fdocuments.in/reader033/viewer/2022052522/5495b07cac7959342e8b4f18/html5/thumbnails/41.jpg)
more elephants● Postgres Pizza BOF:
– Wed Night, 8:30PM, B&G Snooker Hall● pgDay Melbourne: Feb 5th
– 2013.pgday.org.au● pgCon May 21-24 Ottawa● me: www.databasesoup.com
This talk is copyright 2013 Josh Berkus, and is licensed under the Creative Commons Attribution License.Most images are the property of their respective copyright holders, and are used here for purposes of satire.