HOW CAN I BACKUP MY

POSTGRESQL DATABASES?

CUSTOM SCRIPTS, WAL_G, PG_BACKREST,

PG_PROBACKUP, BARMAN??

WHICH ONE SHOULD I USE?

IDENTIFY YOUR NEEDS TO FIND THE RIGHT TOOL FOR YOU!

BARMAN IN ACTION!Giulio Calacoci Senior Developer @ 2ndQuadrant

DataOps 2019 Barcellona

2ndquadrant.com

@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction

ABOUT MYSELF

▸ Open Source passionate since early 2k

▸ Member of the Italian and European PostgreSQL community

▸ Lean and DevOps practitioner

▸ Open Source Developer

▸ Member of the Barman team

▸ Continuous Delivery Architect @2ndQuadrant

▸ 24/7 support engineer @2ndQuadrant

2ndquadrant.com

@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction

OBJECTIVES

• Recovery Point Objective (RPO)

• How much data can I afford to lose?

• Recovery Time Objective (RTO)

• How long will it take me to recover?

2ndquadrant.com

@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction

INCREMENTAL APPROACH

• Keep your goals in mind

• Start (and focus) from disaster recovery

• Incrementally build a DR solution for PostgreSQL (based on barman)

• Evolve from that

2ndquadrant.com

@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction

PLAYGROUND USED IN THIS TALK

https://github.com/2ndquadrant-it/ansible-postgresql-barman-playground

4 machines Vagrant playground environment with Ansible Playbooks for provisioning of PostgreSQL and Barman

Available under GNU GPL 3

After the talk explore the playbook and “play” with it

LET’S START

STEP 1: CREATING THE CLUSTER

2ndquadrant.com

@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction

THE “BEATLES” CLUSTER

4 CentOS pre-configured virtual machines:

• John: PostgreSQL 11 + barman-cli on 192.168.33.11 •Paul: PostgreSQL 11 + barman-cli on

192.168.33.10 •George: Barman 2.8 on

192.168.33.12 •Ringo: Barman 2.8 on 192.168.33.13

2ndquadrant.com

@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction

WHAT TO EXPECT FROM THE PLAYGROUND

• Basic PostgreSQL configuration

• PGDATA initialised with data checksums

• Archive mode enabled

• Syslog destination for logs

• Enable md5 access from 192.168.33.x in pg_hba • Empty database for pgbench

• Paul and John have a running PostgreSQL instance

• SSH key exchange between postgres and barman users

STEP 2: VERIFY THE SETUP

2ndquadrant.com

@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction

GOALS

• Verify that PostgreSQL is up and running on Paul

• Check PostgreSQL configuration

• Generate some data

Paul

PostgreSQL Master

STEP 3: PREPARE POSTGRESQL

2ndquadrant.com

@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction

GOALS

• Prepare PostgreSQL for interacting with barman: • Create 'barman' superuser on Paul • Create 'streaming_barman' replication user on

Paul

STEP 4: OUR FIRST BACKUP

2ndquadrant.com

@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction

GOALS

• Setting up barman • Basic barman concepts • Configuring barman for backing up Paul • Configuring barman for using streaming archive

• First backup using barman

2ndquadrant.com

@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction

ARCHITECTURE

Continuous backup

streaming wal

STEP 5: INCREMENTAL BACKUP

2ndquadrant.com

@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction

GOALS

• Setting up barman to be more space-efficient: • Incremental backups concepts • Incremental backups options • copy • link

STEP 6: REMOTE RECOVERY

2ndquadrant.com

@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction

STARTING ARCHITECTURE

barman recover

Paul

George

John

2ndquadrant.com

@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction

GOALS

• Remote recovery on John

• Setup of John’s new cluster as Paul’s standby

• Setting on-demand WAL fetching

2ndquadrant.com

@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction

FINAL ARCHITECTURE

barman_restore_walPaul

George

John

STEP 7: WAL STREAMING

2ndquadrant.com

@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction

GOALS

• Connect John’s cluster to the Paul’s one

• Setting up WAL streaming

Paul

George

John

WAL streaming

Paul barman_restore_wal

STEP 8: ZERO DATA LOSS

2ndquadrant.com

@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction

GOALS

• Achieving zero data loss (RPO = 0) using the architecture we built so far

• synchronous standby from master and barman

2ndquadrant.com

@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction

ARCHITECTURE

barman_restore_wal

barman recover

Synchronous

ZERO DATA LOSS

STEP 9: GEO-REDUNDANCY

2ndquadrant.com

@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction

GOALS

• Setting up geo-redundancy using Ringo as external barman passive server

• Backing up your backups!

2ndquadrant.com

@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction

ARCHITECTURE

Paul

George

John

WAL streaming

Ringo

barman sync

Data centre 1 (EU) Data centre 2 (USA)

BUT… THERE IS MORE!

2ndquadrant.com

@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction

ARCHITECTURE EVOLUTION (1)

Paul

George

John

WAL streaming

Ringo

barman sync

Data centre 1 (EU) Data centre 2 (USA)

Bert

barman recover

2ndquadrant.com

@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction

ARCHITECTURE EVOLUTION (2)

Paul

George Ringo

barman sync

Data centre 1 (EU) Data centre 2 (USA)

John

barman backupbarman backup

OTHER BARMAN FEATURES

2ndquadrant.com

@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction

BARMAN FEATURES

• Backup frequency (cron for “barman backup”)

• Retention policies

• Redundancy (number of backups)

• Point of Recoverability (based on time)

• Archive command (barman-wal-archive)

• Streaming backup

• Point In Time Recovery (define a target)

•

BARMAN IN THE FUTURE!

2ndquadrant.com

@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction

BARMAN FUTURES

• Direct cloud storage (starting with AWS S3) - 2.9

• barman-cloud-wal-archive • barman-cloud-wal-restore • barman-cloud-backup • barman-cloud-recover

• Integration of the above with Barman - 2.10/3.0?

•Multi-tier setups, based on retention policies:

• Tier 2: Local compressed backups

• Tier 3: Cloud storage

THANK YOU!

2ndquadrant.com

@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction

VIDEOS!

• https://asciinema.org/a/CO270x2NNfBi9vZPS6orM5zRE• https://asciinema.org/a/rz3edZO7RoaqHOrZ2M3oid85x• https://asciinema.org/a/cNdxNdIZQE5xOf1vBVhbM27ft• https://asciinema.org/a/KXRk5NE8DcXaScElXTfXji7vT• https://asciinema.org/a/fuC4xNVGbyfHQZH9UgEShhzYu• https://asciinema.org/a/qEFfCDpPxxzz6cAktbBb92otZ• https://asciinema.org/a/0NEYGKrwUeqtxF94OneWdOR2t• https://asciinema.org/a/2QGItjUKHhYfA70vX3FDIMxfb• https://asciinema.org/a/eG5168n1xLPpFApI9UvvP4jSZ

2ndquadrant.com

@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction

SPECIAL THANKS TO:

• The Beatles!

• The missing members Barman development team

• For Ansible automation:

• Rubens Souza

• Anna Bellandi

• Benjamin Zander for Rule #6:

• https://www.youtube.com/watch?v=M-HG6X6fpBM

• You for surviving/not falling asleep

2ndquadrant.com

@asdmaster @2ndQuad #PostgreSQL #DataOps #Barcellona #BarmanInAction

LICENCE

Attribution 4.0 International (CC BY 4.0)

You are free to:

• Share — copy and redistribute the material in any medium or format

• Adapt — remix, transform, and build upon the material for any purpose, even commercially.

The licensor cannot revoke these freedoms as long as you follow the license terms.