PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference...
Transcript of PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference...
![Page 1: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/1.jpg)
PostgreSQLBackup Strategies
PostgreSQL Conference Europe 2012Prague, The Czech Republic
Magnus [email protected]
PRODUCTS • CONSULTING • APPLICATION MANAGEMENT • IT OPERATIONS • SUPPORT • TRAINING
![Page 2: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/2.jpg)
Replication!● But I have replication!● To multiple nodes!● It's even in the cloud!
![Page 3: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/3.jpg)
What about clustering?● Yeah, pretty much the same
![Page 4: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/4.jpg)
But my SAN is 100% up!● Really?
![Page 5: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/5.jpg)
But my SAN is 100% up!● Really?● No, really?!
![Page 6: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/6.jpg)
Backup planning● Backup interval● Backup retention● Performance impacts
![Page 7: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/7.jpg)
Restore planning● Time spent taking backups usually not
important● Time it takes to restore is critical● Consider multi-stage solutions
![Page 8: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/8.jpg)
PostgreSQL options● Logical backups
● pg_dump● Physical backups
● Filesystem snapshots● pg_basebackup● “Manual” base backups
![Page 9: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/9.jpg)
Logical backups● SQL script dump of schema + data● Restored through SQL commands● Great flexibility● Not the greatest for performance
![Page 10: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/10.jpg)
pg_dump● This is your main tool
● Dumps a single database● Regular PostgreSQL connection● Guarantees consistent snapshot across
database● Single threaded
● (for now..)
![Page 11: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/11.jpg)
pg_dump● Supports multiple output formats
● Always use “custom” format (-Fc)● Compressed by default (-Fc)
● Supports dumping separate objects● For backups, always dump whole
database
![Page 12: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/12.jpg)
pg_dump system impact● Runs regular COPY queries● Uses single backend● Does not ruin PostgreSQL cache
● “ring buffer” strategy used● Can potentially ruin filesystem cache● Writing of dump file causes I/O
![Page 13: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/13.jpg)
pg_dump compression● Compression happens in pg_dump● Can be used for throttling
● Typical “breakpoint” at 3-5● Higher becomes CPU bound● Lower becomes I/O bound
![Page 14: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/14.jpg)
pg_dump ssh tunnel● ssh dbserver "pg_dump -Z9 -Fc -U postgres mydb" > mydb.dump
![Page 15: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/15.jpg)
pg_dump ssh tunnel● ssh dbserver "pg_dump -Z9 -Fc -U postgres mydb" > mydb.dump
● ssh -o "Compression=no" magh.u.bitbit.net "pg_dump -Z9 -Fc -U postgres mydb" > mydb.dump
![Page 16: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/16.jpg)
Restoring from pg_dump● Use pg_restore
● Reads “custom” format dumps● Regular connection
● Full database restore● “Recover from backups”
● Partial database restore● “Create staging env”● “Single table restore”
![Page 17: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/17.jpg)
Restore performance● Regular COPY
● Followed by CREATE INDEX● And ADD CONSTRAINT
● Very slow for large databases!
![Page 18: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/18.jpg)
Restore performance● Use -1 flag● Full restore as single transaction● Enables multiple optimizations
● Particularly if WAL archiving not enabled
● Empty database in case of crash
![Page 19: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/19.jpg)
Restore performance● Restore in parallel sessions
● -j <n>● Each object still in one session● Not compatible with -1
● Need to pick one● -j usually faster
![Page 20: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/20.jpg)
Restore performance● Turn fsync=off
● Last resort● But quite useful
● Don't forget to turn it back on!● (Yes, it happens)
● Don't forget to flush OS caches!● (Yes, you'll get corruption)
![Page 21: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/21.jpg)
Don't forget!● pg_dumpall -g
● Users, groups, tablespaces
![Page 22: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/22.jpg)
Physical backups
![Page 23: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/23.jpg)
Physical backups● PostgreSQL stores database in files● We can backup those files...● No need to parse or query
● Thus faster!● Architecture, version, compile flags and
paths must be identical● Only full cluster backups
![Page 24: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/24.jpg)
Offline backups● Easiest possible way
● Stop PostgreSQL, take backup, start PostgreSQL
● Backup files any way possible● Tar, copy, filesystem snapshot etc
● Not to be ignored...
![Page 25: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/25.jpg)
Simple snapshot backups● Filesystem/SAN snapshots while database
is running● Requires atomic snapshot across all
tablespaces● Including pg_xlog
● Mainly useful in small installations
![Page 26: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/26.jpg)
Online base backups● Non-constrained filesystem level backups● Recoverable in combination with
transaction log● With or without log archive● Provides base for PITR
![Page 27: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/27.jpg)
Online base backups● Integrated base backups
● On top of replication protocol● Enable replication!
● wal_level=archive● max_wal_senders=2
![Page 28: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/28.jpg)
Online base backups● pg_basebackup -U postgres -D backup -P -x
● Requires “enough” WAL to stay around
● Generates complete data directory
![Page 29: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/29.jpg)
Log archiving● As log is generated, send to archive● On restoring, fetch back from archive
● Start from base backup● “Roll forward” through archived log● Stop at any point
![Page 30: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/30.jpg)
Log archiving in PostgreSQL● archive_mode=on
● Starts the log archiver● archive_command=<something>
● “take file x and store it under the name y”
● restore_command=<something>● “give me back the file you stored under
name y”
![Page 31: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/31.jpg)
Log archiving limitations● Always 16Mb segments
● archive_timeout=<n>● Too much or not enough● Replication solves problem in 9.1● 9.2: pg_receivexlog
![Page 32: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/32.jpg)
Base backups for PITR● pg_basebackup without -x● Manual method:
● SELECT pg_start_backup();● <copy files>
– Copy files, e.g. cp/tar– Rsync– SAN snapshots
● SELECT pg_stop_backup();
![Page 33: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/33.jpg)
pg_basebackup system impact
● Reads all data, generates lots of I/O● pg_basebackup single threaded
● This is probablyusually a good thing● Sequential reads● (Optional) compression happens in
pg_basebackup, not server
![Page 34: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/34.jpg)
Restore performance● Depends on “distance to base backup”● Read back all log files, replays
● Generates random writes● Single threaded as well
– (more or less)● Multiple generations of base backups
![Page 35: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/35.jpg)
Backup strategies
![Page 36: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/36.jpg)
Please make backups
![Page 37: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/37.jpg)
How to back up● You definitely want online physical
backups● You almost certainly want PITR● You probably want pg_dump
● If you can afford it
![Page 38: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/38.jpg)
Backup retention● Comes back to business requirements● How far back does it make sense to
restore data?● And at what resolutions?
![Page 39: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/39.jpg)
Log file/base backup● Restore requires base backup + all log
files since with no “holes”● Keep fewer base backups but all logs● Keep fewer logs but more base backups
![Page 40: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/40.jpg)
Backup vs replication● You probably want both● Backups are more important● Replication good for hardware failure● And allows for much shorter service
interruption
![Page 41: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/41.jpg)
Lagged behind replicas● Using file based replication● Introduce delay in the system
● E.g. 1 hour or 12 hours● Roll forward replica instead of restoring
from backups
![Page 42: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/42.jpg)
Testing your backups
![Page 43: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/43.jpg)
Testing your backups● We all know we should● And we seldom do
![Page 44: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/44.jpg)
Use for staging and dev● Restore from backup instead of deploy
from master● Do not automate!
![Page 45: PostgreSQL Backup Strategies - Hagander · PostgreSQL Backup Strategies PostgreSQL Conference Europe 2012 Prague, The Czech Republic Magnus Hagander magnus@hagander.net PRODUCTS •](https://reader034.fdocuments.in/reader034/viewer/2022050918/6028eaefb9ee0637f62a3a66/html5/thumbnails/45.jpg)
Thank you!
Questions?Share your stories!
Twitter: @magnushaganderhttp://blog.hagander.net/[email protected]